A multi-modal model-based multi-interface mobile application evaluation method and device

Abstract

The application relates to a multi-interface mobile application evaluation method and device based on a multi-modal model and belongs to the field of mobile application quality testing.The method comprises the following steps: test item acquisition, information extraction, review information synthesis, information analysis, and result filtering and output, specifically, test link acquisition is completed through a graphical user interface agent technology, accurate information extraction is completed based on a multi-modal large language model and tool calling, user demand analysis and review task planning are completed through multi-agent cooperation, deep consistency analysis is completed based on thought chain reasoning, and finally, a high-quality problem report is output through an intelligent filtering mechanism.The application realizes the automatic extraction, intelligent analysis and systematic evaluation of cross-page information, can efficiently identify information inconsistency problems in a mobile application, improves the pertinence and effectiveness of problem discovery, guarantees the high confidence of test results, reduces the pressure of manual review, and provides technical support for application quality optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mobile application quality testing, in particular to a multi-interface mobile application evaluation method and device based on a multi-modal model. BACKGROUND

[0002] In the current field of mobile application development, as the complexity of applications increases and the iteration speed accelerates, traditional manual quality assurance (QA) methods have been difficult to fully cover all test scenarios. Especially in large-scale applications, due to functional differentiation, team size expansion, and high customization of functions, information content, and business services, inconsistencies between different pages and modules are prone to occur, which violates the consistency principle in user experience design and seriously affects the usability and user experience quality of the application.

[0003] From the perspective of interactive experience, information inconsistency (such as conflicting prices on different pages) can significantly damage user trust and cognitive fluency, thereby disrupting the construction of user mental models, leading to cognitive dissonance and decision paralysis. This experience problem involves deep cognitive consistency and system credibility principles, having a significant negative impact on user trust, task completion rate, and brand perception. Existing testing methods have many limitations: 1. Manual review methods are costly, have low coverage, and are highly subjective, making it difficult to meet the needs of rapid iteration development; 2. Traditional automated testing mainly focuses on functional verification and cannot understand semantic-level information inconsistency issues; 3. Existing automated GUI agent testing mainly focuses on task completion and functional executability, and conducts evaluation at the accessibility level, but lacks deep analysis capabilities for cross-page information consistency; 4. Information consistency detection requires comprehensive understanding capabilities across pages and modalities (text and vision), and relies on deep cognitive understanding of application information architecture, and existing methods have obvious shortcomings in these areas.

[0004] For example, the invention patent with publication number CN119883888A discloses a user interface defect detection method, device, equipment and medium, relating to the field of smart home / smart home technology, which includes: obtaining a user interface image and a standard interface image corresponding to the user interface image, and extracting first features of the standard interface image; determining a first detection rule corresponding to the standard interface image according to the first features of the standard interface image and historical defect data, and recommending the first detection rule to a test end; performing consistency detection on the user interface image and the standard interface image according to a second detection rule fed back by the test end to obtain defect information of the user interface image. This method improves the accuracy and efficiency of interface consistency detection through intelligent recommendation and automatic detection, but it mainly focuses on the comparison between a single interface and a standard interface, and cannot solve the problem of cross-page and cross-process information consistency detection.

[0005] The application No. CN120336183A discloses an application testing method, device, storage medium and program product, which comprises the following steps: controlling a user interface of a to-be-tested application to slide in a specified direction; after the user interface slides in the specified direction, determining whether the interface screenshot before the sliding of the user interface is consistent with the interface screenshot after the sliding of the user interface by using a trained preset model; when the two interface screenshots are inconsistent, determining that the user interface has been slid to a boundary position. The method mainly focuses on the boundary detection of the interface sliding behavior and does not analyze the semantic consistency. Figure One

[0006] The application No. CN120371694A discloses an application No. CN120371694A discloses a user interface automatic assertion method, system and program product based on a multi-modal large model, which comprises the following steps: obtaining a screen image of a to-be-tested UI and a text instruction describing an expected state; inputting the image and the text instruction into a multi-modal large model configured to jointly process the image and the text; analyzing whether the actual UI state in the image meets the expectation of the text instruction by using the image-text joint understanding capability of the multi-modal large model; and generating a final assertion result based on the model determination result. Although the method applies the multi-modal large model to verify the UI state, it mainly focuses on the expected matching of a single interface state and lacks the systematic detection capability for the cross-page information flow and semantic consistency.

[0007] Therefore, there is an urgent need for a technical solution capable of automatically and systematically detecting the cross-page information consistency problem of a mobile application. SUMMARY

[0008] The application aims to provide a multi-interface mobile application evaluation method and device based on a multi-modal model, which aims to solve the problem that the prior art only focuses on a single interface and cannot automatically and systematically detect the cross-page information consistency problem of a mobile application.

[0009] The technical solution adopted by the application to solve the technical problem is as follows:

[0010] A multi-interface mobile application evaluation method based on a multi-modal model, comprising the following steps:

[0011] Test item collection: executing the interaction of a specified test link through a graphical user interface agent, collecting interface screenshots and user interaction history records in the interaction process;

[0012] Information extraction: extracting structured information from the interface screenshots by using an information extractor combining a large language model and a tool call, and generating structured data containing multiple information groups;

[0013] ​Review information synthesis: based on structured data, using a large language model to analyze user information needs, establishing cross-interface information association graph based on parsed user information needs, and combining preset information consistency checkpoints to investigate cross-interface information content, generating review files;

[0014] Information analysis: based on the generated review files, different analysis agents are used according to the nature of the review task, and the reasoning process and analysis results of the information consistency problem are output by using the thinking chain reasoning mode for parallel analysis;

[0015] Result filtering and output: construct heuristic rules based on keywords to preliminarily screen the analysis results, and then filter through the filter based on search enhancement generation technology, output the final problem report, and complete the evaluation.

[0016] In one embodiment, in the test item collection, the user interaction history record includes:

[0017] Interface summary information, a simplified natural language description of each interface generated by the graphical user interface agent, which extracts the key functional areas and main information content of the interface, providing a basis for subsequent interaction decisions of the graphical user interface agent;

[0018] Interaction operation record, used to record the specific operation behavior of the graphical user interface agent on each interface, each interaction operation record at least contains operation type and operation object.

[0019] In one embodiment, in information extraction, the information extractor based on a large language model combined with tool calling performs structured information extraction on the interface screenshot, including:

[0020] Macro semantic understanding of the interface screenshot by the large language model, performing context-aware block division to form information groups, and parsing the content and visual semantics of the information groups to realize basic information extraction, wherein each information group contains a unique identifier, block position, information summary, and a list of component lists;

[0021] Then the large language model calls external tool set to verify and enhance the accuracy of the extracted basic information, and completes the extraction of structured information.

[0022] In one embodiment, the external tool set includes one or more of an optical character recognition tool, a color picker tool, or a pixel-level visual analysis tool, which is used to supplement and confirm the extracted information; wherein the supplemented and confirmed information includes color values, accurate text content, and pixel-level visual features.

[0023] In one embodiment, in the information extraction, the organizational framework of the structured data comprises three layers, namely, user interaction history record, information group summary, and specific information content of the information group;

[0024] The user interaction history record comprises interaction path, interface summary, and operation record, for providing a macro business process view;

[0025] The information group summary comprises overall description and block division of each information group in the interface, for providing a meso interface structure view;

[0026] The specific information content of the information group comprises detailed information and visual features of each component in the information group, for providing a micro information detail view.

[0027] In one embodiment, in the review information synthesis, the construction process of the cross-interface information association graph comprises:

[0028] According to the user interaction history record, a complete operation sequence of the user between interfaces is extracted as a process skeleton of the cross-interface information association graph; meanwhile, based on the structured information unit defining the specific business information contained in each interface as a semantic node library of the cross-interface information association graph;

[0029] Based on the process skeleton and the semantic node library, an operation path of the user between interfaces is extracted, and the information groups involved in the path are located, forming an information-enhanced operation sequence;

[0030] The association between the information groups is inferred through two-dimensional logic, the association comprising establishing a process association with a front-back sequence relationship according to the sequence of the information groups in the operation path of the user, and analyzing the business content carried by the information groups to determine whether they point to the same business object or describe the same business attribute;

[0031] All information groups are taken as nodes, and the inferred process association and semantic association are taken as edges, to establish the cross-interface information association graph.

[0032] In one embodiment, in the review information synthesis, the preset information consistency check points comprise numerical information contradiction, text information contradiction, visual style contradiction, information loss, and information increase;

[0033] The numerical information contradiction comprises numerical conflict and specification conflict; wherein the numerical conflict refers to numerical information inconsistency of a product displayed in different interfaces, including numerical contradiction of time, price, quantity, or geographical location distance; the specification conflict refers to product specification information contradiction in different display positions, including non-uniform unit or coexistence of fuzzy description and accurate description;

[0034] The text information contradiction includes using different text expressions for the same semantics, contradictory texts before and after, and ambiguous texts.

[0035] The visual style contradiction includes different visual styles for the same level information and similar visual styles for different levels of information. The different visual styles for the same level information refer to using different visual styles for interface elements with the same content or the same function. The similar visual styles for different levels of information refer to using the same or similar visual styles for information of different levels or different states, resulting in unclear information level distinction.

[0036] The information loss refers to the disappearance of key information during page jumping.

[0037] The information increase refers to the sudden appearance of new important information in the later stage of the process.

[0038] In one embodiment, the result filtering and outputting, the filtering by the filter based on the retrieval enhancement generation technology, includes:

[0039] Receiving the preliminary filtered analysis result, and performing abstraction processing on the analysis result to remove specific business information to extract problem pattern features, and generating a query vector;

[0040] In the pre-constructed low-quality output knowledge base, a number of historical low-quality output cases most similar to the query vector are searched through vector similarity retrieval;

[0041] Based on the retrieved similar cases, it is determined whether the current analysis result belongs to a known low-quality output pattern;

[0042] If the similarity of the current analysis result to the known low-quality output pattern exceeds a preset threshold, it is filtered; if it does not exceed, the analysis result is retained as a high-confidence output, and a final problem report is obtained.

[0043] In one embodiment, the construction process of the low-quality output knowledge base includes:

[0044] Collecting invalid outputs identified by preliminary screening settlement and low-quality problem reports found by manual review;

[0045] Using a large language model to summarize and classify the collected low-quality problem reports, and removing specific business information in the cases and retaining the abstract features of the problem patterns in the process;

[0046] Performing text embedding processing on the problem patterns that have been abstracted, and storing them in the knowledge base for retrieval.

[0047] In addition, to achieve the above object, the application also provides a multi-modal model-based multi-interface mobile application evaluation device, which is used to realize the multi-modal model-based multi-interface mobile application evaluation method, and comprises:

[0048] A test item acquisition module is configured to collect interface screenshots and user interaction history records in an interaction process by executing interactions of a specified test link through a graphical user interface agent.

[0049] An information extraction module is configured to extract structured information from the interface screenshots based on an information extractor combining a large language model and a tool call, and generate structured data comprising a plurality of information groups.

[0050] An examination information synthesis module is configured to analyze user information requirements using a large language model based on the structured data, establish a cross-interface information association graph based on the parsed user information requirements, and check cross-interface information content in combination with preset information consistency checkpoints to generate an examination file.

[0051] An information analysis module is configured to use different analysis agents according to the nature of the examination task based on the generated examination file, and perform parallel analysis using a thinking chain reasoning mode to output a reasoning process and analysis results of information consistency problems.

[0052] A result filtering module is configured to preliminarily filter analysis results based on heuristic rules based on keywords, and then filter through a filter based on search enhancement generation technology to output a final problem report and complete the evaluation.

[0053] The application has the following effects by adopting the above technical solutions:

[0054] (1) Systematic cross-page information consistency detection capability: the application automatically collects a plurality of interface information in a user interaction process through a graphical user interface agent, extracts structured information and performs cross-page association analysis in combination with a multi-modal large language model, realizes systematic detection of full-process information consistency of a mobile application, and overcomes the limitation of traditional methods that can only perform single-page or functional testing.

[0055] (2) The application analyzes user requirements in depth, establishes a cross-interface information association graph, systematically locates a cross-interface information set to be compared for subsequent consistency review, and establishes an information consistency checkpoint system, which can comprehensively cover typical information consistency problem types in a mobile application, making consistency detection more close to actual user experience and improving the pertinence and effectiveness of problem discovery.

[0056] (3) Through a multi-level information extraction and analysis architecture, progressive analysis from a macroscopic to a microscopic level is realized, and the accuracy of problem positioning is improved.

[0057] (4) By multiple analysis agents in parallel, combined with thought chain reasoning and knowledge base-based result filtering mechanism, false positive rate is effectively reduced, high confidence of test results is ensured, and artificial review pressure is reduced.

[0058] (5) The whole evaluation process realizes full automation from test item collection to result output, can quickly adapt to different types of mobile applications and test scenarios, significantly improves test efficiency and coverage, and reduces labor cost. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows.

[0060] Figure 1 is the framework diagram of the multi-interface mobile application evaluation method based on the multi-modal model provided by the embodiment of the present application.

[0061] Figure 2 is the flowchart of the multi-interface mobile application evaluation method based on the multi-modal model provided by the embodiment of the present application.

[0062] Figure 3 is the structural schematic diagram of the multi-interface mobile application evaluation device based on the multi-modal model provided by the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical scheme and advantages of the present application more clear and obvious, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0064] As shown in Figure 1 , the overall flowchart of the multi-interface mobile application evaluation method based on the multi-modal model provided by the embodiment of the present application. The present application realizes the automatic identification and evaluation of the cross-page information consistency problem in mobile application through multi-modal large language model technology. The method includes five core steps of test item collection, information extraction, review information synthesis, information analysis and result filtering, forming a complete evaluation closed loop.

[0065] S1, test item collection: through the graphical user interface agent to execute the interaction of the specified test link, collect the interface screenshots and user interaction history records in the interaction process. Specifically as follows:

[0066] As shown in Figure 2As shown, this step completes the automatic interaction of the specified test link through an intelligent graphical user interface agent (GUI), and collects interface screenshots and user interaction history records in the interaction process. This step serves as the data collection link of the method of the present application, providing a basic data source for subsequent information consistency evaluation.

[0067] In this embodiment, the system can complete the automatic collection of test links for various application scenarios (such as e-commerce shopping, social media, financial services, etc.). This embodiment takes clothes shopping in e-commerce shopping as an application scenario. The test link is defined in the form of a task target, and the GUI agent can autonomously complete the full-process interaction from the starting page to the target page and record the complete user interaction history.

[0068] The user interaction history record mainly contains two types of core information: interface summary information and interaction operation record. The interface summary information describes the key content of the page in natural language form, and the interaction operation record details the user's operation behavior and object on the interface. These two types of information together with the interface screenshots form a structured test data set, providing a complete data basis for subsequent cross-interface information consistency analysis.

[0069] Through this intelligent test item collection method, the system can efficiently simulate the interaction path of real users and obtain complete interface information sequences, thereby providing comprehensive and accurate raw data input for subsequent information extraction and consistency evaluation.

[0070] S2, information extraction: based on the information extractor combining large language models and tool calls, structured information extraction is performed on the interface screenshots to generate structured data containing multiple information groups. Specifically as follows:

[0071] The core of this step is to build a two-stage information extraction architecture of "semantic understanding-precision enhancement". This architecture combines the macro semantic understanding ability of multi-modal large language models with the micro data precision of special tool sets, achieving high-quality and structured analysis of interface information, and providing a reliable data basis for subsequent cross-interface consistency evaluation.

[0072] First Stage: Macro Semantic Understanding Based on Expert Role Simulation. In this stage, through precise prompt tuning, a multi-modal large language model is configured as a senior UI (user interface) analysis expert agent. The core task of this expert agent is not simply to identify elements, but to deconstruct the interface from the perspective of business logic and user experience, mainly in the following two aspects: Context-aware Chunking: The model goes beyond traditional segmentation methods based on geometric location and adopts an intelligent grouping strategy based on semantic relevance. It aggregates logically related elements in the interface (such as product pictures, titles, prices, and promotion labels) into independent "information groups." For example, in a list page containing multiple product cards, the model will explicitly indicate in the description of each information group that "the product card with the model wearing a white T-shirt in the picture" or "the product card with the title 'Pure Cotton Short Sleeve Round Collar Shirt'." This division ensures the business integrity of the information and provides stable anchors for subsequent cross-interface tracking of specific objects (such as the same product's information on different pages).

[0073] Multi-dimensional Visual Semantics Analysis: The model not only extracts literal information such as text and numerical values, but also deeply analyzes the business meanings implied by the visual forms it carries. Significant visual features for deep analysis include but are not limited to: bold font, italic font, text color, strikethrough, underline, background color, font size, corner size, transparency, border style, etc. The model identifies these visual attributes to infer the importance and business meaning of the information. For example: the model can understand that a strikethrough on the price usually means "original price," while highlighted or specially colored text represents "promotion information" or "key tips." This decoding ability of visual language makes the extracted information go beyond the original data and contain rich business status and information level annotations.

[0074] When dealing with interfaces containing multiple similar objects (such as product lists, SKU selectors, and coupon lists), the model has the ability to associate scattered information with specific objects. This ability ensures that the extracted data is not only accurate but also has a clear attribution relationship, avoiding confusion and mismatch of information. This association recognition capability is particularly important when dealing with product lists. When multiple product cards appear on a product list page, each card contains product pictures, prices, titles, and promotion labels. The model needs to identify which price, picture, and description belong to which product. The model distinguishes similar objects through descriptive features, such as "the product with the model wearing a white T-shirt in the picture, priced at ¥99" and "the product with the model wearing a black T-shirt in the picture, priced at ¥129."

[0075] The second stage is micro-precision enhancement based on function calling. To overcome the inherent limitations of large language models in pixel-level precision, the architecture introduces a function calling mechanism as the key second stage. After completing macro semantic understanding, the model can intelligently identify which information needs higher precision verification and actively call external specialized tool sets for confirmation and enhancement.

[0076] The workflow of this collaborative mode is as follows: the large language model performs macro semantic understanding on the interface screenshot, performs context-aware block division to form information groups, and analyzes the content and visual semantics of the information groups to achieve basic information extraction and generate a call request. Then the system executes tools such as optical character recognition (OCR) model, color picker or pixel-level visual model, and returns the accurate results (such as accurate text content, hexadecimal color code, element pixel size, etc.) to the model for supplement and confirmation of the extracted information. The supplemented and confirmed information includes color values, accurate text content and pixel-level visual features. The model finally integrates the information of the two stages and outputs a fully structured data with macro semantic understanding and micro data precision. Through this two-stage architecture, the invention not only solves the problem that traditional automation technology cannot understand interface semantics, but also makes up for the shortcoming of relying solely on large language models that cannot guarantee absolute data accuracy.

[0077] Finally, each interface is converted into a three-layer information architecture consisting of "user interaction history record", "information group summary" and "component specific information". The user interaction history record contains interaction path, interface summary and operation record, providing a macro business process view.

[0078] The information group summary contains the overall description and block division of each information group in the interface, providing a meso interface structure view.

[0079] The specific information content of the information group contains detailed information and visual features of each component in the information group, providing a micro information detail view. This three-layer information architecture provides a solid foundation for multi-level, cross-interface information consistency analysis in subsequent steps.

[0080] S3, review information synthesis: based on the structured data, use the large language model to analyze the user information demand, establish a cross-interface information association graph based on the parsed user information demand, and combine the preset information consistency checkpoints to investigate cross-interface information content, and generate a review file.

[0081] The innovation of this step is to build a multi-agent collaborative, hierarchical and progressive cognitive synthesis framework. This framework aims to simulate the complete mental model of expert consistency review: first, understand the user's core intent and information needs from the macro process; second, based on the intent, build cross-interface review assumptions at the meso level; finally, package the assumptions with precise data at the micro level to form a structured analysis file. This process makes full use of the three-layer information architecture generated in the previous stage.

[0082] First stage: user intent simulation based on user interaction history. This stage focuses on answering "why review". This invention uses the first level of information architecture (user interaction history) to deeply simulate the user's cognitive and decision-making process in the interaction process through an expert user agent (Expert User Agent). This agent analyzes the user's operation sequence, not only to record behavior, but also to infer the underlying goals and key information needs at different decision nodes.

[0083] The core function of this agent is that it can distinguish and identify information types that have a high impact on user decision-making (such as price, delivery time, discount conditions, etc.), giving subsequent review tasks priority and targeting based on the user's perspective. This makes consistency detection no longer a indiscriminate full comparison, but rather an intelligent review focused on protecting core user experience.

[0084] For example, in the clothes shopping scenario, the user's interaction history includes the following key nodes: browsing multiple product cards on the product list page → clicking on a white T-shirt to enter the product detail page → viewing product parameters and user reviews → selecting size and color → clicking the "Buy Now" button → entering the order confirmation page → viewing delivery information and coupons → clicking "Submit Order" to enter the payment page. By analyzing this operation sequence, the user's decision-making process is inferred, and special attention is paid to key turning points and information acquisition nodes in the user interaction path. Combined with the inference mechanism of differentiated user needs, it is determined which information has a decision-making impact on specific types of users. This ability enables the agent to go beyond superficial information lists and identify key information that is scenario-specific; combined with the decision-making impact assessment framework, it analyzes the user's behavior changes before and after obtaining a certain type of information to determine the impact of the information, for example, if the user sees a "limited-time 5% off" label on the product list page and immediately clicks into the detail page, it indicates that the promotional information has a high decision-making impact; if the user significantly extends the time spent on the detail page after viewing the return policy, it indicates that the protection information has a high experience impact.

[0085] Second stage: Review task construction based on information group association. After understanding the user's intention, this stage focuses on answering "what to review" and "where to review". The invention uses the second level of information architecture (information group overview) to construct specific review tasks through a task planning agent.

[0086] The core capability of this agent is to construct a cross-interface information association graph. It tracks the flow and evolution of "information groups" describing the same business object between different interfaces according to the user's actual interaction path, and associates them. Then, according to the pre-set five consistency check point system (covering numerical, text, visual, etc.), the agent automatically generates structured review suggestions (Analysis Proposal) for these associated information groups. Each suggestion is a specific, executable, and verifiable consistency check hypothesis. Specifically:

[0087] According to the user's interaction history, the complete operation sequence of the user between interfaces is extracted as the process skeleton of the cross-interface information association graph; at the same time, based on the structured information unit that defines the specific business information contained in each interface as the semantic node library of the cross-interface information association graph;

[0088] Based on the process skeleton and semantic node library, the user's operation path between interfaces is extracted, and the information groups involved in the path are located, forming an information-enhanced operation sequence;

[0089] For example: In the clothes shopping scenario, the agent tracks the user's complete operation path: the user browses the second product card on the product list page → clicks to enter the product detail page to view price information → selects the M size specification to view the corresponding price → adds the product to the shopping cart to view the product information on the shopping cart page → enters the order confirmation page to view the order amount → views the payment amount on the payment page.

[0090] The agent identifies all information groups related to the same product in this path and establishes an association relationship. For example: the product card on the product list page and the product information area on the product detail page have semantic association, describing the basic information of the same product;

[0091] The price information on the product detail page and the size price in the SKU selector have semantic association, describing the price of the same product in different specifications;

[0092] The product information on the shopping cart page, the product list on the order confirmation page, and the order details on the payment page have both process and semantic association, describing the information presentation of the user-selected product in different transaction stages.

[0093] This kind of mapping information provides clear object positioning for subsequent expert analysts, ensuring that the same object information across interfaces can be systematically identified and compared.

[0094] By inferring the association between information groups in two dimensions, the association includes establishing a flow association with a front-back sequence relationship according to the position of the information group on the user operation path, and analyzing the business content carried by the information group to determine whether it points to the same business object or describes the semantic association of the same business attribute; for example, in the clothes shopping scenario, the business side may explicitly require checking "product price", "delivery fee", "return policy" and other information that must be accurately displayed according to legal and platform rules; the user side analysis identifies "coupon use conditions", "estimated arrival time", "inventory quantity" and other information that has a significant impact on user decision-making. After integrating the two dimensions, the information types such as price information and delivery information that appear in the two-dimensional list at the same time are the core inspection objects, and information types such as coupon use restrictions that are only concerned in the user dimension are supplementary inspection objects.

[0095] All information groups are taken as nodes, and the inferred flow association and semantic association are taken as edges to establish a cross-interface information association graph.

[0096] Subsequently, according to the preset information consistency check points, structured review suggestions are automatically generated for these associated information groups:

[0097] Checkpoint one: numerical information contradiction, including two subtypes of numerical conflict and specification conflict.

[0098] The numerical conflict refers to the inconsistency of numerical information displayed by the product on different pages, including but not limited to: time value contradiction (such as "tomorrow delivery" on the list page and "3-5 days delivery" on the detail page), price value contradiction (such as "¥99" on the list page and "¥109" on the detail page), quantity value contradiction (such as "inventory sufficient" on the detail page and "only 1 left" when ordering), and geographic location distance value contradiction (such as "3.2 kilometers away" on the list page and "5.1 kilometers away" on the map page). In the clothes shopping scenario, if the product card on the product list page displays a promotional price of "¥99", but after the user clicks into the product detail page, the price is found to be "¥109", or after selecting the M size specification, the price becomes "¥119" without a price difference between different specifications being indicated on the list page, then a price value conflict is constituted.

[0099] The specification conflict refers to the contradiction of product specification information in different display positions, including but not limited to: unit inconsistency (such as the detail page shows "500g", the parameter table shows "0.5kg", although the value is equivalent, but the expression way is inconsistent), fuzzy description and accurate description coexist (such as the list page shows "about 3 days to arrive", the order page shows "expected delivery on December 25", the time range of the two may not match). In the clothes shopping scene, if the specification parameter of the product detail page shows "fabric composition: 95% cotton", and the product description text also mentions "100% pure cotton material", it constitutes a specification conflict.

[0100] Checkpoint two: contradictory information in the text, including three subtypes of using different text expressions for the same semantics, contradictory text before and after, and ambiguous text.

[0101] Using different text expressions for the same semantics refers to the use of different word descriptions for the same concept or function in different interfaces, which may lead to user understanding difficulties or doubts. In the clothes shopping scene, if the product list page shows "7-day no reason return", the product detail page shows "support 7-day worry-free return", and the order confirmation page shows "return guarantee: one week return", although the semantics are the same, the difference in expression may make the user doubt whether the specific policy content is consistent.

[0102] The contradiction before and after the text includes two cases of commitment invalidation contradiction and positive and negative expression difference. Commitment invalidation contradiction refers to the commitment or explanation made on the previous page being invalid or appearing opposite expression on the subsequent page. In the clothes shopping scene, if the product detail page promises "free shipping over 99 yuan", but after the user adds a product priced at ¥99 to the shopping cart, the order confirmation page shows "shipping fee ¥10", it constitutes a commitment invalidation contradiction. The positive and negative expression difference refers to the appearance of opposite expressions of the same information on different pages, such as the product detail page marking "no support for 7-day no reason return", and the order confirmation page showing "enjoy worry-free return service".

[0103] The ambiguity of the text refers to the text description with multiple understanding ways, which may lead to user's wrong expectation. In the clothes shopping scene, if the product detail page describes "free shipping" but does not specify whether there are regional or amount restrictions, the user may understand it as unconditional free shipping, while the actual situation may be "free shipping over 99 yuan" or "free shipping only in some areas", such ambiguous expression is easy to cause user dissatisfaction.

[0104] Checkpoint three: visual style contradiction, including two subtypes of different styles of the same level information and similar styles of different levels of information.

[0105] The different styles of the same level information refer to the interface elements with the same content or the same function using different visual styles, which destroys the consistency rules of the interface. In the clothes shopping scenario, if the first product's promotional price in the product list page is displayed using red 20-point bold font, and the second product's promotional price is displayed using orange 18-point normal font, although both are promotional price information, the difference in visual style may make the user mistakenly think that the two products have different promotional strength or importance.

[0106] For another example, if the "buy now" button of the product detail page uses a red background with white text, and the "go to checkout" button of the shopping cart page uses an orange background with black text, although both are main operation buttons, the difference in style will weaken the user's awareness of the continuity of the operation process.

[0107] The similar styles of different levels of information refer to the information of different levels or different states using the same or similar visual styles, which leads to unclear information level distinction. In the clothes shopping scenario, if the product original price (invalid reference price) in the product detail page uses gray 14-point font, and the secondary explanation text (such as "subject to the actual object") in the product parameters also uses gray 14-point font, it will be difficult for the user to distinguish the importance difference between the price information and the explanation information.

[0108] For another example, if the promotional label "limited time 5 off" and the ordinary product attribute label "pure cotton" use the same visual style (the same background color, font size, and border style), the promotional information will be weakened, and the user's perception of the preferential information will be affected.

[0109] Checkpoint four: information loss refers to the disappearance of key information with the interface jump in the user task flow, which leads to the user's inability to obtain the information in the subsequent decision-making link. The key information includes but is not limited to at least one of the following: coupon information, time-limited activity information, delivery benefit information, and delivery state information.

[0110] In the clothes shopping scenario, typical information loss cases include: 1. Coupon loss: the user sees a certain product marked "available for full-reduced coupon" on the product list page, and after clicking into the product detail page, the coupon information disappears, and the user cannot understand the specific coupon use conditions and the discount strength; 2. Time-limited activity loss: the user sees the activity countdown "limited time 5 times, remaining 2 hours" on the product detail page, and after joining the shopping cart, the countdown information disappears on the shopping cart page, the user cannot perceive the activity urgency, and may miss the preferential period; 3. Delivery benefit loss: the user sees the "member exclusive free shipping" mark on the product detail page, but the benefit description disappears on the order confirmation page, resulting in the user being unable to confirm whether to enjoy the free shipping benefit before payment; 4. Delivery status loss: the user sees the "present stock fast delivery, expected tomorrow" delivery promise on the product detail page, but only "standard delivery" is displayed on the order confirmation page without specifying the specific time limit, and the user cannot verify whether the time limit promise is valid. The core of the information loss problem is that the user makes a decision (such as selecting the product, joining the shopping cart) based on the information on the previous page, but cannot confirm the information again in the subsequent link, resulting in the loss of decision basis and the reduction of user trust.

[0111] Checkpoint five: information increase refers to the sudden appearance of important limiting conditions that have not been mentioned before the key decision node of the user task flow, affecting the decision effectiveness of the user. The key decision node includes but is not limited to the order confirmation page (bill page) or payment page, and the important limiting condition includes but is not limited to at least one of the product delivery restriction, group purchase use restriction, and benefit enjoyment restriction.

[0112] In the clothes shopping scenario, typical information increase cases include: 1. Delivery restriction increase: the user does not see any delivery restriction information on the product list page and the detail page, and normally selects the product and adds it to the shopping cart, but suddenly prompts "the product does not support delivery to your address" or "remote area needs to pay an additional ¥20 freight" on the order confirmation page. At this time, the user has invested a lot of time cost, and this sudden restriction will seriously affect the user experience; 2. Increase of discount restrictions: the user sees the "reduce 20 yuan for 99 yuan" discount information on the product detail page, selects two products with a total price of ¥118, but suddenly prompts "this coupon is only for purchasing a single product" or "this discount is not available with member discounts" when using the discount on the order confirmation page. This type of previously unmentioned usage restriction causes the user's discount expectations to fall through; 3. Payment restriction increase: the user does not see any payment method restriction information on the previous page, but suddenly discovers "the product does not support cash on delivery" or "additional 3% handling fee for credit card payment" on the payment page. The sudden appearance of such payment restrictions may cause the user to abandon the purchase. The core of the information increase problem is that the user does not receive complete decision-making information in the early stages of the decision-making path, and only discovers important restriction conditions at the critical node just before completing the transaction. At this time, the user has invested a lot of time and psychological cost, and the sudden appearance of restrictions will cause strong frustration and damage to trust.

[0113] The third stage: analysis case packaging based on accurate content. Finally, this stage focuses on "how to review", that is, to provide complete input for subsequent in-depth analysis. The system integrates the "review suggestions" generated in the previous stage with the third level of information architecture (specific information content), and automatically packages it into an independent review file (Review File) through a script.

[0114] Each review file is a complete analysis case, which not only contains the review instructions to be executed, but also encapsulates all the context required to execute the instructions: including macro-intentions derived from user interaction history records, meso-review guidelines generated by task planning agents, and all micro-detail data extracted from related "information groups".

[0115] Through this cognitive synthesis framework, the invention systematically transforms raw and scattered multi-interface information into a series of structured analysis tasks with clear goals, complete data, and rich context, laying a solid foundation for high-quality, automated reasoning analysis in step S4.

[0116] S4, based on the generated review file, different analysis agents are used according to the nature of the review task, and the thought chain reasoning mode is used for parallel analysis, outputting the reasoning process and analysis results of the information consistency problem.

[0117] This step, as the core of the system's automated reasoning and judgment, is responsible for executing the structured review tasks generated in the previous step. Its innovation lies in efficient task distribution and interpretable reasoning mechanisms.

[0118] This step receives the "review files" from step S3 and intelligently distributes them to analysis agents (Analysis Agent) with corresponding expertise according to the nature of the review tasks (such as numerical contradictions, visual style contradictions, etc.), for parallel processing, thereby achieving optimal utilization of resources and specialization of analysis.

[0119] The core working mode of each analysis agent is Chain-of-Thought (CoT) reasoning. This mechanism forces the agent to first generate a complete and clear logical reasoning process before giving the final judgment. This "process-first" output mode ensures that every problem discovered has transparent explainability and traceability, allowing human auditors to clearly verify the basis of their judgments. Finally, the agent outputs a structured problem report, which not only contains problem details but also includes a complete reasoning chain, realizing an intelligent closed loop from "finding problems" to "arguing problems".

[0120] S5, Result filtering and output: Constructing keyword-based heuristic rules to preliminarily filter analysis results, and then using search enhancement generation technology-based filters for screening, outputting the final problem report, and completing the evaluation.

[0121] To ensure the high fidelity and high credibility of the final output report, this step introduces an innovative, self-optimizing dual-filter quality assurance framework.

[0122] First, efficient heuristic filtering. The system first uses keyword-based fast heuristic rules, including but not limited to: "inconsistency", "contradiction", "conflict", "difference", "error", "problem", "risk", etc. Through keyword detection, the system can identify model outputs that do not contain problem report keywords. This stage aims to efficiently filter out obviously invalid or format error outputs as the first quality threshold, reducing the pressure of subsequent fine filtering.

[0123] Second round: intelligent filtering based on retrieval enhancement. The core innovation of this step is an intelligent filtering agent based on a retrieval-augmented generation (RAG) technology. The invention constructs and maintains a dynamic "low-quality output knowledge base", the initial data of which comes from invalid outputs identified in the rule filtering stage and low-quality problem reports found through manual review. All outputs are archived in the knowledge base after manual review to ensure the accuracy and representativeness of the knowledge base content. The knowledge base is managed using a large language model, which summarizes and classifies low-quality outputs. When processing low-quality outputs, the system removes specific business information (such as product names, specific price values, page IDs, etc.), retains the abstract features of the problem pattern, and then performs text embedding processing. This abstraction process allows the knowledge base to identify common patterns of low-quality outputs without being limited by specific business scenarios.

[0124] Typical low-quality output patterns include but are not limited to:

[0125] 1. Over-speculation: the model makes excessive speculations in the absence of sufficient information, reporting problems that do not exist;

[0126] 2. Misunderstanding business logic: the model lacks domain knowledge and misjudges normal business logic as inconsistent problems (such as believing that different specifications of goods should have the same price);

[0127] 3. Fuzzy positioning: the problem report lacks precise page and information group positioning, making it impossible to reproduce the problem;

[0128] 4. Inadequate reasoning: the reasoning process lacks logical support, and the basis for problem judgment is unclear;

[0129] Repeated reporting: repeated reporting of the same problem from different angles increases the burden of manual review.

[0130] When an analysis result enters this filtering stage, the filtering agent abstracts the analysis result to be detected, removes specific business information, extracts problem pattern features, and generates a query vector. Next, the system performs vector similarity retrieval in the low-quality output knowledge base to find a number of historical low-quality output cases most similar to the result to be detected;

[0131] Then, the filtering agent combines the retrieved similar cases and the current analysis result to determine whether the current result belongs to the low-quality output patterns identified in the knowledge base. If the similarity exceeds the preset threshold (set to 0.8 in this embodiment), it is determined to be a low-quality output and is filtered out;

[0132] Finally, the system keeps the high-confidence bug reports that are not filtered out, and outputs them as the final test results.

[0133] The low-quality output knowledge base supports continuous iterative optimization. The sources of the knowledge base updates include:

[0134] 1. Automatic accumulation: the invalid outputs identified by the rule filtering are automatically archived to the knowledge base;

[0135] 2. Manual labeling: the testers can manually add the newly discovered low-quality output patterns to the knowledge base when they review the filtering results;

[0136] 3. Business customization: the testers can manually access specific non-concerned content to avoid repeated reporting of problems that repeatedly occur but have no optimization plan in the short term.

[0137] The self-optimization feature of the framework is reflected in that the knowledge base can be continuously iterated through manual feedback and the addition of new samples. This enables the filtering system to continuously learn and gradually adapt to the business logic of specific applications, thereby continuously improving the accuracy of filtering and ensuring that the final delivery to developers and testers is high-confidence and high-value bug reports.

[0138] So far, the five core steps of the information consistency evaluation method for multi-interface mobile applications provided by the present application have been completed. The method completes test link collection through GUI proxy technology, accurate information extraction based on multi-modal large language model and tool invocation, user demand analysis and review task planning through multi-agent collaboration, deep consistency analysis based on thought chain reasoning, and finally outputs high-quality problem reports through intelligent filtering mechanism. The entire process realizes the automated identification and evaluation of mobile application information consistency problems, providing effective technical support for application quality optimization.

[0139] Please refer to Figure 3 , based on the above method, the present application also provides a multi-interface mobile application evaluation device based on a multi-modal model, the multi-interface mobile application evaluation system is used to realize the multi-interface mobile application evaluation method, comprising:

[0140] A test item collection module is used to execute the interaction of the specified test link through the graphical user interface agent, and collect the interface screenshots and user interaction history records in the interaction process;

[0141] An information extraction module is used to extract the structured information of the interface screenshots through the information extractor combined with the large language model and tool invocation, and generate the structured data containing multiple information groups;

[0142] The review information synthesis module is configured to analyze user information requirements using a large language model based on structured data, establish a cross-interface information association graph based on the parsed user information requirements, and check cross-interface information content in combination with preset information consistency checkpoints to generate a review file;

[0143] The information analysis module is configured to use different analysis agents according to the nature of the review task and adopt a thinking chain reasoning mode for parallel analysis based on the generated review file, and output a reasoning process and analysis result of the information consistency problem;

[0144] The result filtering module is configured to construct heuristic rules based on keywords to preliminarily screen the analysis result, and then screen the analysis result through a filter based on search enhancement generation technology to output a final problem report and complete the evaluation.

[0145] It should be noted that the multi-modal model-based multi-interface mobile application evaluation method and device provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the multi-modal model-based multi-interface mobile application evaluation method, which will not be repeated here.

[0146] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or terminal. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or terminal including the element.

[0147] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0148] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can improve or modify the application according to the above description, and all such improvements and modifications shall fall within the scope of the claims of the present application.

Claims

1. A method for evaluating multi-interface mobile applications based on a multimodal model, characterized in that, Includes the following steps: Test item collection: The interaction of the specified test link is executed through the graphical user interface agent, and the interface screenshots and user interaction history are collected during the interaction process; Information Extraction: An information extractor based on a combination of large language model and tool calls performs structured information extraction on interface screenshots, generating structured data containing multiple information groups; Review information synthesis: Based on structured data, large language models are used to analyze user information needs. Based on the parsed user information needs, a cross-interface information association graph is established. Combined with preset information consistency checkpoints, cross-interface information content is checked and a review document is generated. The process of constructing the cross-interface information association graph in the review information synthesis includes: Based on user interaction history, the complete operation sequence of the user between interfaces is extracted as the process skeleton of the cross-interface information association graph. Simultaneously, structured information units defining the specific business information contained in each interface are used as the semantic node library of the cross-interface information association graph. Based on the process skeleton and semantic node library, the user's operation path between interfaces is extracted, and the information groups involved in the path are located to form an information-enhanced operation sequence. The association between information groups is inferred through two-dimensional logic. This association includes establishing process associations with sequential relationships based on the order of information groups on the user's operation path, and analyzing the business content carried by the information groups to determine whether they point to the same business object or describe the same business attribute. All information groups are used as nodes, and the inferred process associations and semantic associations are used as edges to establish the cross-interface information association graph. Information Analysis: Based on the generated review documents, different analysis agents are used according to the nature of the review task, and parallel analysis is performed using a thought chain reasoning mode to output the reasoning process and analysis results of the information consistency problem; Results filtering and output: Construct keyword-based heuristic rules to initially screen the analysis results, then use a filter based on search enhancement generation technology to further filter the results, and finally output a problem report to complete the evaluation.

2. The multi-interface mobile application evaluation method based on a multimodal model according to claim 1, characterized in that, The user interaction history recorded in the test item collection includes: Interface overview information, a simplified natural language description of each interface generated by the graphical user interface agent. The simplified natural language description extracts the key functional areas and main information content of the interface, providing a basis for the graphical user interface agent's subsequent interaction decisions. Interaction logs are used to record the specific actions performed by the graphical user interface agent on each interface. Each interaction log contains at least the action type and the action object.

3. The multi-interface mobile application evaluation method based on a multimodal model according to claim 1, characterized in that, In the information extraction process, the information extractor based on a combination of a large language model and tool calls performs structured information extraction from the interface screenshot, including: The large language model performs macro-semantic understanding of the interface screenshot, performs context-aware block division to form information groups, and analyzes the content and visual semantics of the information groups to extract basic information. Each information group contains a unique identifier, block location, information overview and a list of its components. Next, the large language model calls an external toolset to verify and enhance the accuracy of the extracted basic information, thus completing the extraction of structured information.

4. The multi-interface mobile application evaluation method based on a multimodal model according to claim 3, characterized in that, The external toolset includes one or more of optical character recognition tools, color picker tools, or pixel-level visual analysis tools, which are used to supplement and confirm the extracted information; wherein, the supplemented and confirmed information includes color values, precise text content, and pixel-level visual features.

5. The multi-interface mobile application evaluation method based on a multimodal model according to claim 4, characterized in that, In information extraction, the structured data organization architecture consists of three layers: user interaction history, information group overview, and specific information content of the information group. The user interaction history includes interaction path, interface overview and operation record, which is used to provide a macro view of the business process; The information group overview includes an overall description and block division of each information group in each interface, outlining the content, visual features and location features of each information group, and is used to provide a meso view of the interface structure. The specific information content of the information group includes detailed information and significant visual features of each component within the information group, used to provide a microscopic view of information details.

6. The multi-interface mobile application evaluation method based on a multimodal model according to claim 1, characterized in that, In the process of reviewing information synthesis, the preset information consistency checkpoints include: numerical information contradictions, textual information contradictions, visual style contradictions, information loss, and information additions; The numerical information contradictions include numerical conflicts and specification conflicts; wherein, the numerical conflicts refer to inconsistencies in the numerical information of the product displayed on different interfaces, including numerical contradictions in timeliness, price, quantity, or geographical distance; the specification conflicts refer to contradictions in the product specification information displayed in different locations, including inconsistent units or the coexistence of vague descriptions and precise descriptions. The contradictions in the text include using different text to express the same meaning, contradictions within the text, and ambiguity in the text. The visual style contradictions include different styles for information at the same level and similar styles for information at different levels; wherein, different styles for information at the same level means that interface elements with the same content or the same function use different visual styles; and similar styles for information at different levels means that information at different levels or in different states uses the same or similar visual styles, resulting in unclear distinction between information levels. The information loss refers to the disappearance of key information when the page redirects; The information is added when new and important information suddenly appears later in the process.

7. The multi-interface mobile application evaluation method based on a multimodal model according to claim 1, characterized in that, In the result filtering and output, the filtering performed using a filter based on retrieval enhancement generation technology includes: Receive the analysis results after preliminary screening, and abstract the analysis results to remove specific business information in order to extract problem pattern features and generate query vectors; In a pre-built low-quality output knowledge base, vector similarity retrieval is used to find several historical low-quality output cases that are most similar to the query vector. Based on the retrieved similar cases, determine whether the current analysis results belong to a known low-quality output pattern; If the similarity between the current analysis result and a known low-quality output pattern exceeds a preset threshold, it is filtered out; otherwise, the analysis result is retained as a high-confidence output, resulting in the final problem report.

8. The multi-interface mobile application evaluation method based on a multimodal model according to claim 7, characterized in that, The construction process of the low-quality output knowledge base includes: Collect reports of invalid outputs identified during the initial screening and settlement process, as well as reports of low-quality issues discovered during manual review; The collected low-quality problem reports are summarized and categorized using a large language model. In this process, specific business information in the cases is removed, while the abstract features of their problem patterns are retained. The abstracted problem pattern is then embedded into text and stored in a knowledge base for retrieval.

9. A multi-interface mobile application evaluation device based on a multimodal model, characterized in that, The multi-interface mobile application evaluation system is used to implement the multi-interface mobile application evaluation method according to any one of claims 1-8, including: The test item collection module is used to execute the interaction of the specified test link through the graphical user interface agent, and collect the interface screenshots and user interaction history during the interaction process; The information extraction module, based on an information extractor that combines a large language model with tool calls, extracts structured information from screenshots of the interface and generates structured data containing multiple information groups. The review information synthesis module is used to analyze user information needs based on structured data using a large language model, establish a cross-interface information association graph based on the parsed user information needs, and check the cross-interface information content in conjunction with preset information consistency checkpoints to generate review documents. The information analysis module is used to analyze generated review documents, using different analysis agents according to the nature of the review task, and employing a thinking chain reasoning mode for parallel analysis, outputting the reasoning process and analysis results for information consistency issues; The results filtering module is used to construct keyword-based heuristic rules to initially screen the analysis results, and then use filters based on search enhancement generation technology to further filter the results, outputting the final problem report and completing the evaluation.

Images