A method and device for training a multi-modal large model for user experience diagnosis

Abstract

This specification provides a training method and apparatus for a multimodal large-scale model for user experience diagnosis. The method includes: acquiring a first type of sample data and corresponding sample labels; inputting the first type of sample data into the multimodal large-scale model to obtain the output result of the multimodal large-scale model, the output result including the inference process and the prediction conclusion; determining the accuracy of the output result based on the difference between the prediction conclusion and the sample labels; determining a reward value based on the accuracy of the output result and the number of transitional markers during the inference process, wherein, when determining the reward value, for output results whose accuracy meets a preset accuracy requirement, the reward value decreases as the number of transitional markers increases, and for output results whose accuracy does not meet the preset accuracy requirement, the reward value increases as the number of transitional markers increases; and adjusting the parameters of the multimodal large-scale model based on the reward value using a reinforcement learning algorithm. This method can improve the accuracy of multimodal large-scale models in diagnosing user experience.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and specifically, to a training method and apparatus for a multimodal large model for user experience diagnosis. Background Technology

[0002] With the rapid development of internet technology, the user interface (UI) has become a key factor influencing the competitiveness of digital products. Large internet platforms, with their application ecosystems encompassing massive amounts of pages and third-party mini-programs, and frequent feature iterations, result in user experience issues being characterized by their large scale, high degree of concealment, and long-tail distribution. Traditional user experience assessments primarily rely on manual review, requiring expert teams to scan interface screenshots component by component and check each item against the specification manual. This approach is not only time-consuming and labor-intensive—a single page review often takes tens of minutes—but also difficult to achieve scalable coverage. More importantly, many experience issues are highly concealed; for example, the close button is visually weakened in induced activation designs, or the authorization checkbox is checked by default. These problems are often only passively discovered after user complaints cause brand damage, lacking effective means of pre-emptive interception and automated inspection.

[0003] To address the inefficiency and high cost of manual review, the industry has attempted to introduce automated detection technologies. Early solutions were mostly based on Document Object Model (DOM) tree parsing, extracting element positions, attributes, and hierarchical relationships, and combining this with rule engines for detection. However, this type of method can only handle structured data and cannot identify visual experience defects, such as insufficient color contrast or misleading visual design. Furthermore, it is difficult to adapt to complex scenarios such as dynamic rendering, Canvas drawing, and mini-programs, and the rule maintenance cost is extremely high. Another common approach is to train independent, dedicated small models for specific experience problems, such as pop-up detection models or occlusion detection models. This approach results in a linear increase in the number of models with the addition of rules. Each model makes independent decisions, making it difficult to handle complex scenarios that require integrating visual, textual, and interaction logic. Moreover, adding new rules requires retraining the models, resulting in long iteration cycles and failing to meet rapidly changing business needs.

[0004] In recent years, Multimodal Large Language Model (MLLM) has provided a new path for automated experience evaluation. MLLM can process multiple modal data such as images and text simultaneously, and by simulating human cognitive processes, it can comprehensively evaluate the layout, color, usability, and other dimensions of UI design.

[0005] However, existing methods based on multimodal large models often prioritize the accuracy of the final prediction results during training, lacking explicit constraints on the simplicity and computational efficiency of the reasoning process. As a result, when generating Chain of Thought (CoT), the model tends to produce lengthy and repetitive reasoning paths, accompanied by numerous templated self-reflection or error-correction statements. This pseudo-deep reasoning, which is "superficially complex but with limited information gain," not only increases the overhead of generating invalid tokens and computation but may also amplify reasoning instability and the risk of illusion. This redundancy further increases single-sample inference latency, making it difficult to meet the demands of large-scale real-time detection. This reasoning pattern makes it difficult for the model to achieve both high accuracy and high response speed simultaneously with limited computational resources (such as in lightweight model deployment scenarios).

[0006] Therefore, there is a need for a new training mechanism for multimodal large models for user experience diagnosis that can compress output length, reduce inference latency, and maintain or even improve the accuracy of user experience diagnosis. Summary of the Invention

[0007] This specification provides a training scheme for a multimodal large model for user experience diagnosis, so that the trained large model can simultaneously guarantee inference efficiency and inference accuracy during inference.

[0008] In a first aspect, embodiments of this specification provide a training method for a multimodal large model for user experience diagnosis, comprising: acquiring a first type of sample data and corresponding sample labels, wherein the first type of sample data includes at least visual modal data of the user interface, and the sample labels are used to indicate the evaluation result of the user interface based on preset user experience rules; inputting the first type of sample data into the multimodal large model to obtain the output result of the multimodal large model, wherein the output result includes an inference process and a prediction conclusion, the inference process being a thought chain including multiple intermediate inference steps, the inference process including several transition markers, the transition markers being used to indicate the transitions in the inference process, and the prediction conclusion being used to characterize the prediction evaluation result of the user interface based on preset user experience rules; determining the accuracy of the output result based on the difference between the prediction conclusion and the sample labels; determining a reward value based on the accuracy of the output result and the number of transition markers in the inference process, wherein, when determining the reward value, for output results whose accuracy meets the preset accuracy requirements, the reward value decreases as the number of transition markers increases, and for output results whose accuracy does not meet the preset accuracy requirements, the reward value increases as the number of transition markers increases; and adjusting the parameters of the multimodal large model based on the reward value using a reinforcement learning algorithm.

[0009] In one implementation, the first type of sample data includes positive sample data and negative sample data. Positive sample data refers to sample data that does not conform to preset user experience rules, while negative sample data refers to sample data that conforms to preset user experience rules. Obtaining negative sample data from the first type of sample data includes: for each candidate negative sample in the initial negative sample set, performing multiple sampling predictions using an evaluation model to obtain multiple sampling results. The sampling results are used to characterize the classification result of the evaluation model on whether the candidate negative sample conforms to the preset user experience rules; calculating a consistency index of the multiple sampling results, which is used to characterize the uncertainty of the classification result of the evaluation model on the candidate negative sample; comparing the consistency index with a preset threshold. If the consistency index is lower than the preset threshold, the candidate negative sample is determined to be negative sample data in the first type of sample data.

[0010] In one implementation, the first type of sample data includes positive sample data and negative sample data. Positive sample data refers to sample data that does not conform to preset user experience rules, while negative sample data refers to sample data that conforms to preset user experience rules. Obtaining positive sample data from the first type of sample data includes: performing enhancement processing on each candidate positive sample in the initial positive sample set to obtain enhanced candidate positive samples. The enhancement processing includes pixel-level enhancement and / or spatial-level enhancement. Pixel-level enhancement includes brightness adjustment and color dithering, while spatial-level enhancement includes affine transformation and perspective transformation. The candidate positive samples and the enhanced candidate positive samples are used as positive sample data in the first type of sample data.

[0011] In one embodiment, the method further includes: acquiring a second type of sample data and corresponding sample labels, wherein the second type of sample data is used to train the user interface understanding ability of the multimodal large model, the second type of sample data includes at least visual modal data of the user interface, and the corresponding sample labels are used to characterize at least one of the semantic content, element position, or interactive action of the user interface; inputting the second type of sample data into the multimodal large model to obtain the prediction result output by the multimodal large model, wherein the prediction result is used to characterize at least one of the predicted semantic content, predicted element position, or predicted interactive action of the user interface; and determining a reward value based on the difference between the prediction result and the sample labels.

[0012] In one implementation, determining the reward value based on the difference between the prediction result and the sample label includes: identifying the task type to which the currently input second type of sample data belongs, the task type including at least one of the following: semantic understanding task, visual localization task, action prediction task; matching the corresponding target reward function from a preset set of reward functions according to the task type, and calculating the reward value through the target reward function based on the difference between the prediction result and the sample label; wherein, when the task type is a semantic understanding task, the target reward function is configured to calculate the reward value based on the text similarity between the predicted semantic content in the prediction result and the semantic content in the sample label; when the task type is a visual localization task, the target reward function is configured to calculate the reward value based on the position matching degree between the predicted element position in the prediction result and the element position in the sample label; when the task type is an action prediction task, the target reward function is configured to calculate the reward value based on the behavior matching degree between the predicted interactive action in the prediction result and the interactive action in the sample label.

[0013] In one implementation, before inputting the first type of sample data into the multimodal large model to obtain the output of the multimodal large model, the method further includes: supervising the training of the multimodal large model using a supervised fine-tuning dataset; the supervised fine-tuning dataset includes the first type of sample data, the corresponding sample labels, and the associated inference path; wherein, the inference path includes a logical deduction process based on preset user experience rules, and the logical deduction process includes at least: identifying whether there are target elements in the user interface that violate the preset user experience rules, determining the reasons for the violation, and referencing the terms of the preset user experience rules on which it is based.

[0014] In one implementation, the first type of sample data further includes text modal data and / or interaction data of the user interface, wherein the interaction data includes a sequence of user actions on the user interface, and the sequence of actions includes at least one of click location, page jump path and dwell time.

[0015] Secondly, embodiments of this specification provide a method for user experience diagnosis based on a multimodal large model, wherein the multimodal large model is trained based on any of the methods in the first aspect, and the method includes: acquiring user operation data, the user operation data including at least visual modal data of the user interface; inputting the user operation data into the multimodal large model, the multimodal large model evaluating the user experience of the user using the user interface based on preset user experience rules, and obtaining a diagnostic result, the diagnostic result being used to indicate whether the user interface conforms to the preset user experience rules.

[0016] Thirdly, embodiments of this specification provide a training apparatus for a multimodal large model for user experience diagnosis, comprising: an acquisition module for acquiring a first type of sample data and corresponding sample labels, wherein the first type of sample data includes at least visual modal data of the user interface, and the sample labels are used to indicate the evaluation results of the user interface based on preset user experience rules; and an inference module for inputting the first type of sample data into the multimodal large model to obtain the output results of the multimodal large model, wherein the output results include an inference process and a prediction conclusion, the inference process being a thought chain including multiple intermediate inference steps, and the inference process including several transition markers, the transition markers being used to indicate transitions in the inference process. The system comprises four modules: a prediction module, a judgment module, and a reward module. The prediction conclusion characterizes the prediction evaluation result of the user interface based on preset user experience rules; a judgment module determines the accuracy of the output result based on the difference between the prediction conclusion and the sample labels; a reward module determines the reward value based on the accuracy of the output result and the number of transition labels during the inference process, wherein, when determining the reward value, for output results whose accuracy meets the preset accuracy requirements, the reward value decreases as the number of transition labels increases, and for output results whose accuracy does not meet the preset accuracy requirements, the reward value increases as the number of transition labels increases; and a parameter tuning module adjusts the parameters of the multimodal large model based on the reward value using a reinforcement learning algorithm.

[0017] Fourthly, embodiments of this specification provide a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in either the first or second aspect.

[0018] In the scheme provided in this specification, when training a multimodal large model based on reinforcement learning, the reward value is determined by combining the number of transition tags and the accuracy of the inference result during the inference process of the large model's output. Specifically, for output results that meet the accuracy requirements, the reward value decreases as the number of transition tags increases; while for output results that do not meet the accuracy requirements, the reward value increases as the number of transition tags increases.

[0019] This asymmetric reward mechanism adjusts model parameters through reinforcement learning algorithms, enabling the model to gradually learn an adaptive inference strategy during training, achieving a balance between simplicity and sufficiency in inference. Specifically, when facing correctly identifiable UI problems, the model tends to reduce unnecessary intermediate inference steps and transitional markers, thereby generating simpler outputs, effectively reducing the computational load and time consumption of a single inference, and improving diagnostic efficiency. Conversely, when facing complex, error-prone, or difficult-to-judge UI scenarios that lead to initial prediction errors, the model is incentivized to generate more transitional markers and more detailed inference steps to increase the possibility of correcting biases, thus helping to improve diagnostic accuracy on difficult samples. In this way, this solution can maintain or even improve the accuracy of multimodal large-scale models in diagnosing user experience while compressing redundant outputs and reducing inference latency, achieving a balance between inference efficiency and diagnostic accuracy. Even with limited computing resources (such as in lightweight model deployment scenarios), multimodal large-scale models trained using this solution can simultaneously achieve high accuracy and high response speed. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a user interface that can be applied to one of the embodiments of this specification;

[0022] Figure 2 This is a flowchart of a training method for a multimodal large model for user experience diagnosis, as described in the embodiments of this specification.

[0023] Figure 3 This is a schematic diagram of a two-stage training process for a multimodal large model used for user experience diagnosis, as described in the embodiments of this specification.

[0024] Figure 4 This is a flowchart of a method for user experience diagnosis based on a multimodal large model, as described in the embodiments of this specification.

[0025] Figure 5 This is a schematic diagram of the structure of the training device for a multimodal large model used for user experience diagnosis in the embodiments of this specification. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0027] To facilitate understanding of the solutions in this manual, some terms used in this manual will be explained below.

[0028] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, enabling them to generalize. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0029] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0030] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025), and will not be repeated here.

[0031] A multimodal large language model refers to a large-scale neural network model capable of jointly processing, understanding, and generating data from two or more different modalities, such as text, images, video, and audio. In this specification, the multimodal large language model is primarily used for the unified encoding and comprehensive analysis of visual modal data, text modal data, and interaction data of user interfaces. Visual modal data can include screenshots and screen recordings; text modal data can include page text, button text, and prompts; and interaction data can include user click paths, page navigation order, and dwell time. By jointly modeling multiple modalities, this model can identify user experience problems that are difficult to detect with a single modality.

[0032] A thought chain is a reasoning technique where, before outputting a final conclusion, the model first generates several intermediate reasoning steps to simulate the human process of step-by-step analysis and deduction. In this specification, the thought chain is used to guide the model to perform step-by-step analysis around visual elements, textual semantics, and interaction logic in the user interface, thereby improving the accuracy and interpretability of identifying complex experience problems. The thought chain can take the form of intermediate processes such as step-by-step judgment, conditional analysis, evidence induction, or conclusion deduction.

[0033] User experience (UX) refers to the overall feeling and comprehensive evaluation formed by users when using a product, system, or service. UX typically encompasses multiple dimensions such as usability, understandability, operational efficiency, emotional satisfaction, visual comfort, and interactive smoothness. In this specification, user experience issues include, but are not limited to, design flaws that affect normal user use and perceived quality, such as misleading interfaces, cumbersome operations, information obstruction, visual degradation, default selections, and unreasonable pop-up interference.

[0034] ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation based on Longest Common Subsequence) is a natural language processing metric used to evaluate the similarity between generated text and reference text. This metric measures the consistency in content order and expressive structure between the generated and reference texts by calculating the longest common subsequence (LCS). Compared to methods based on fixed phrase matching, ROUGE-L can reflect the similarity of the overall text structure to a certain extent. In this specification, ROUGE-L can be used to evaluate the degree of matching between the reasoning process, diagnostic conclusion, or explanatory text generated by the model and the labeled reference answer.

[0035] User journey (or clickstream) refers to the operational path and behavioral trajectory a user follows while completing a task. A user journey typically includes information such as page navigation sequence, click locations, click order, scrolling behavior, dwell time, and return actions, describing the interaction process between the user and the interface. In this specification, user journeys can be derived from real user operation logs, automated test scripts, or task simulations. By analyzing user journey information, it is possible to identify user experience issues such as misleading paths, redundant steps, hidden critical functions, or abnormal interruptions during the user's operation.

[0036] As mentioned earlier, user experience issues encountered by users when using the user interface are often subtle, such as the visually weakened close button in an incentive-based activation design, or the default selection of the authorization checkbox. Figure 1The user interface shown has the text "Synchronize and Enable XX Priority Payment" in a smaller font size and lighter color compared to other text on the page. The button text reads "Enable and Receive," and there is no option to disable XX Priority Payment. When a user attempts to receive a red envelope, they may not see the "Synchronize and Enable XX Priority Payment" text and accidentally enable it, or they may have to enable XX Priority Payment simultaneously to receive the red envelope, resulting in a poor user experience. Therefore, this solution uses a multimodal large model to diagnose the user interface and determine if any experience issues exist.

[0037] However, during the process of diagnosing user interfaces using a multimodal large model, the inventors discovered that the model can easily disguise its reasoning depth by generating lengthy intermediate thought processes. This leads to uncontrollable output length, decreased reasoning efficiency, and affects the stability and accuracy of the final diagnostic results. Specifically, existing technologies typically regard words or markers in the model output that indicate transitions, reflections, or corrections, such as "but," "however," "wait," or phrases like "Let me reconsider the problem," "However, I need to consider another possibility," or "Let me think about it," as representations of intermediate reasoning, self-correction, or reflection. In some studies, the appearance of such transition words is even considered a sign that the model has achieved an "insight moment," meaning that the model can actively correct its original judgment and further approach the correct conclusion during the reasoning process. However, the inventors found that the frequency of reflective behavior is not necessarily positively correlated with the correctness of the model's output. On the contrary, when reflective transition words appear too frequently, the model often does not demonstrate stronger reasoning ability but is prone to repeated negation, self-wavering, and ineffective wandering, resulting in a large consumption of computational resources.

[0038] To verify the above issues, the inventors conducted experimental analysis, statistically analyzing the number of transition markers T that appeared in the model during reasoning for each training or evaluation sample. These transition markers included, but were not limited to, words indicating transitions, reflections, or corrections such as "but," "however," and "wait." Experimental results showed that for samples with incorrect predictions, the average number of transition markers reached 14.38; while for samples with correct predictions, the average number of transition markers was only 2.48, with the former being approximately six times the latter. Furthermore, among the incorrect samples, approximately 31.5% of the samples had more than three transition markers. These samples typically exhibited characteristics such as repeatedly overturning previous conclusions, wavering between multiple candidate judgments, and long but ineffective reasoning chains. In contrast, the reasoning paths of correct samples were generally more direct, coherent, and stable, exhibiting only moderate self-correction.

[0039] The experimental results above demonstrate that in user experience problem diagnosis tasks, more model reflection is not necessarily better. Moderate reflection helps the model identify biases in previous inferences and further improves its analytical capabilities for complex problems; however, when the number of reflections exceeds a reasonable range, the model is prone to entering a state of "overthinking." In this state, the model constantly questions existing reasonable judgments, performs inefficient or even random searches in the semantic space, and fails to effectively converge to the correct conclusion, instead easily deviating from the correct direction. In other words, existing reinforcement learning training methods, lacking constraints on intermediate reasoning processes, easily induce the model to form "pseudo-deep reasoning" by piling up transition words and lengthy steps, which increases token overhead and is detrimental to improving diagnostic performance.

[0040] Based on this, this scheme designs an asymmetric transition label reward mechanism in the reinforcement learning stage of multimodal large models to balance the sufficiency and simplicity of the model's reasoning process. Specifically, for correctly predicted samples, an "overthinking prevention" mechanism is adopted, causing the reward value to decrease linearly with the increase of the number of transition labels. A lower bound on the reward can also be set to avoid the model being over-penalized for the necessary reasoning required to process complex samples. In this way, the model is guided to use more refined and stable reasoning paths when it can correctly complete the task. For incorrectly predicted samples, an "exploration incentive" mechanism is adopted, causing the reward value to increase linearly with the increase of the number of transition labels, to encourage the model to conduct more thorough analysis and attempts before reaching the correct conclusion.

[0041] Through the aforementioned asymmetric reward design, this scheme can differentiate and adjust the model's reflective behavior based on the model's prediction results: on the one hand, it suppresses invalid and redundant reasoning in correct samples and reduces overthinking; on the other hand, it preserves the necessary exploration space in incorrect samples, enhances the model's reflective and error-correcting capabilities in complex user experience problems, thereby improving the reasoning efficiency, diagnostic accuracy, and actual deployment performance of the multimodal large model in user interface experience diagnostic tasks.

[0042] The training process of this scheme will be described below with reference to specific embodiments.

[0043] Figure 2 This is a flowchart of a training method for a multimodal large model for user experience diagnosis, as described in the embodiments of this specification. This method can be applied to any device, platform, or device cluster with computing and processing capabilities, including steps 201-203 as shown below.

[0044] In step 201, the first type of sample data and the corresponding sample labels are obtained.

[0045] The first type of sample data includes at least visual modal data of the user interface, typically represented by screenshots or screen recordings, which intuitively reflect the layout, color scheme, element hierarchy, and visual style of the user interface. In addition to visual information, the first type of sample data may also include at least one of textual modal data and interaction data of the user interface to provide richer contextual information. Textual modal data includes all visible text content displayed on the interface, such as button text, titles, body text, and prompts. This textual information helps the model understand the semantic functions and business logic of interface elements. Interaction data records the user's behavioral trajectory or movement on the interface, specifically including operation sequences. These sequences encompass metrics such as the coordinates of click locations, page navigation paths, and dwell time in specific areas or pages. By integrating visual, textual, and interaction data, the first type of sample data can construct a multimodal input space combining static presentation and dynamic behavior, enabling the model to comprehensively understand the user experience from both perceptual and behavioral perspectives.

[0046] The sample labels corresponding to the first type of sample data are used to indicate the evaluation results of the user interface based on preset user experience rules. Preset user experience rules refer to a predefined set of standards for measuring interface quality, which can cover multiple dimensions such as usability, efficiency, and reliability. The specific form and content of the sample labels can be flexibly configured according to actual training needs; this embodiment does not impose strict limitations on this.

[0047] In one implementation, the sample labels can be categorical labels used to identify the evaluation results of the user interface in a specific user experience diagnostic task. Specifically, the sample labels can be binary labels, such as marked "compliant" or "non-compliant," to directly indicate whether the interface meets preset user experience rules; or they can be multiple-choice labels used to determine the correct answer from a set of predefined structured options, such as options A, B, or C, where each option corresponds to a specific interface state or defect type (e.g., "no modal pop-up exists," "modal pop-up lacks an explicit close control," or "modal pop-up has an explicit close control"). This label design based on structured options can cover a variety of specific experience problem scenarios such as pop-up stacking, obscuring the close button, and inconsistent content.

[0048] In another, more granular implementation, sample labels can specifically identify the violated user experience rules or problem categories, such as explicitly labeling "continuous pop-ups interfering with user operation," "inconsistent information between advertisements and landing pages," or "non-blocking bubbles obscuring clickable hotspots." Furthermore, sample labels can include spatial location information, indicating the coordinates or bounding box range of the specific UI element violating the rule within the interface, thus guiding the model to focus on specific visual areas for diagnosis. Sample labels can also be further linked to a multi-dimensional user experience evaluation system, representing the compliance status of the user interface in different dimensions such as efficiency, trustworthiness, or usability. This diverse label design enables the model to learn multi-level diagnostic capabilities during training, ranging from macro-level compliance judgments to micro-level defect localization and classification, providing accurate and rich supervisory signals for subsequent generation of predictive conclusions that include reasoning processes.

[0049] This embodiment does not limit the specific method of obtaining the first type of sample data and the corresponding sample labels. For example, after collecting the first type of sample data, manual annotation or voting by multiple large models can be used to obtain sample labels. During the collection of the first type of sample data, in order to ensure that the trained multimodal large model has broad applicability and robustness, the source of the sample data covers diverse operating environments and display states. Specifically, the first type of sample data includes user interfaces from different operating systems, such as application interfaces on iOS and Android platforms, to cover the differences in UI design under different system specifications. At the same time, the sample data also includes the performance of the same application or page in different theme modes, including light mode and dark mode, to test the model's ability to recognize visual elements such as color contrast and text readability in different backgrounds. In addition, the first type of sample data also covers screenshots of interfaces with various device orientations, including portrait mode and landscape mode, as well as the adaptation of different screen resolutions and aspect ratios. By introducing these multi-source data with significant distribution differences, we can comprehensively evaluate and improve the model's generalization ability when facing different platform specifications, visual styles and layout structures, prevent the model from overfitting to a specific single scenario, and ensure that it can maintain stable diagnostic performance in the actual complex and ever-changing production environment.

[0050] In some embodiments, sample labels may only include annotations of the final evaluation results based on preset user experience rules, without annotations of the intermediate reasoning process. This design aims to reduce data annotation costs and enhance the model's autonomous reasoning capabilities. Specifically, annotators or large model annotation personnel only need to determine whether the first type of sample data has user experience issues or whether it conforms to preset user experience rules, thereby generating result labels indicating "correct" or "incorrect," "compliant" or "non-compliant," without manually writing complex thought processes or detailed reasoning steps. By simplifying the annotation focus from "how to reason" to "whether the result is correct," the workload of human intervention and subjective bias are significantly reduced.

[0051] Under this training mechanism, the multimodal large model is configured to autonomously explore reasoning paths from input data to the final conclusion through reinforcement learning algorithms. During training, the model attempts to generate various intermediate reasoning steps and transition labels. The system provides verifiable reward signals based on the consistency between the model's output prediction and the sample labels. When a reasoning path generated by the model leads to a correct conclusion consistent with the sample label, the path receives a positive reward and is thus reinforced in parameter updates; conversely, when a reasoning path leads to an incorrect conclusion contrary to the sample label, the path receives a negative reward or a lower reward value and is thus suppressed in parameter updates. This result-verification-based autonomous learning mechanism not only reduces reliance on external expert knowledge but also enables the model to discover reasoning logic and feature associations that are not pre-defined by human experts but are equally effective or even superior. This allows the model to gradually optimize its internal reasoning structure during long-term training, improving the accuracy and interpretability of diagnosis.

[0052] In step 202, the first type of sample data is input into the multimodal large model to obtain the output results of the multimodal large model.

[0053] The output includes the reasoning process and the predicted conclusion. The reasoning process is a thought chain comprising multiple intermediate reasoning steps. This thought chain aims to simulate the step-by-step cognitive process of human experts diagnosing user experience. The reasoning process includes several transition markers, which indicate shifts in the reasoning process and can also represent reflection or correction, such as words or phrases in natural language like "but," "however," "wait a minute," "let me reconsider the problem," "however, I need to consider another possibility," or "but let me think about it." These transition markers not only connect different stages of reasoning but also explicitly express the model's logical jumps and self-verification behavior during the reasoning process. The predicted conclusion characterizes the final predictive evaluation result of the user interface based on preset user experience rules, such as determining whether the interface is compliant, the specific type of violation, or the severity.

[0054] In one embodiment, based on the thought chain reasoning technology, the structured diagnostic reasoning process generated by the model can be decomposed into four consecutive steps: perception, analysis, reasoning, and conclusion. For example, in the perception step, the model first performs preliminary identification of the input user interface visual modal data and text modal data, extracting key UI elements and their attributes, such as button positions, colors, text content, and layout structure. In the analysis step, the model combines the extracted features with contextual information for in-depth interpretation, identifying spatial relationships, semantic connections, and potential interaction logic between elements. In the reasoning step, the model calls a preset user experience rule base or activates model parameters storing the user experience rule base, compares the current interface's feature state with the rule requirements, and handles complex or ambiguous scenarios through logical transitions guided by transition markers. For example, when the initial judgment conflicts with the rules, the model triggers a reflection mechanism through markers such as "However" or "Let me reconsider," re-examining visual evidence or adjusting the preconditions for rule application. Finally, in the conclusion step, the model synthesizes the results of the aforementioned perception, analysis, and reasoning to generate the final prediction conclusion.

[0055] In some embodiments, prompt words can be input together with the first type of sample data into a multimodal large model. The prompt words are used to prompt the model to perform a user experience diagnostic task based on the first type of sample data. For example, when the user experience diagnostic task is to diagnose whether a pop-up window affects the user experience, the prompt words can be as follows:

[0056] Core task: Based solely on a single screenshot of the user interface, determine whether the pop-up window provides the user with a clear option to close it.

[0057] Options include:

[0058] A. No modal pop-up windows appeared.

[0059] B. The pop-up window does not provide a clear option to close it.

[0060] C. The pop-up window has a clear close control option.

[0061] Please analyze the provided user interface screenshots and output your evaluation results in the format $\boxed{X}$ (where X is one of A, B, or C).

[0062] In step 203, the accuracy of the output results is determined based on the difference between the prediction conclusion and the sample label.

[0063] This embodiment does not limit the specific comparison method for determining accuracy, and can be flexibly selected according to the data form of the prediction conclusion and sample labels and the task requirements.

[0064] In one implementation, when both the prediction conclusion and the sample label are structured option identifiers (such as "A", "B", "C") or explicit classification labels, an exact matching method can be used. That is, it is directly determined whether the identifier of the prediction conclusion is completely consistent with the identifier of the sample label. If they are consistent, it is determined to be accurate; otherwise, it is determined to be inaccurate.

[0065] In another implementation, when the predicted conclusion is expressed as a natural language description and the sample label is a semantic definition, semantic matching can be used. This involves calculating the semantic similarity between the predicted conclusion text and the sample label description text, or using a pre-trained language understanding model to determine whether they express the same meaning. If the semantic consistency meets a preset threshold, it is considered accurate. Furthermore, other logical judgment methods such as rule matching and keyword extraction can be combined to determine accuracy. Regardless of the comparison method used, the core purpose is to objectively quantify the degree of consistency between the model output and the true annotations, thereby generating an accuracy metric for subsequent reward calculation and ensuring that the feedback signals during reinforcement learning truly reflect the model's performance on the current diagnostic task.

[0066] In step 204, the reward value is determined based on the accuracy of the output and the number of transition markers during the reasoning process.

[0067] The number of transition markers refers to the total number of words or phrases (such as "but," "however," "wait a minute," etc.) indicating transitions, reflections, or corrections included in the reasoning process of the output result. This embodiment uses an asymmetric reward mechanism to determine the reward value. Specifically, for output results that meet the preset accuracy requirements (i.e., correct results where the predicted conclusion is consistent with the sample label), the reward value decreases as the number of transition markers increases. This means that when the model correctly diagnoses a user experience problem, the system tends to encourage simpler and more direct reasoning paths and penalize lengthy and unnecessary intermediate steps, thereby suppressing the model's behavior of "faking" reasoning depth by piling up invalid transition markers and improving reasoning efficiency. For outputs whose accuracy does not meet the preset accuracy requirements (i.e., erroneous results where the predicted conclusion is inconsistent with the sample label), the reward value increases with the number of transition labels. Alternatively, the penalty value applied to erroneous results decreases with the number of transition labels. This means that when the model diagnoses errors, the system tends to give relatively higher rewards (or lower penalties) to outputs that contain more reasoning steps and reflection processes. This encourages the model to explore and experiment more deeply when faced with complex or difficult samples, avoiding abandoning deep thinking due to excessive pursuit of brevity. This helps the model correct biases and improve its ability to handle difficult examples in subsequent training.

[0068] Furthermore, to prevent the model from using the aforementioned reward mechanism for speculative optimization, this embodiment also sets upper and lower bound constraints on the reward value. Specifically, for output results whose accuracy does not meet the preset accuracy requirement, although the reward value increases with the number of transition markers, it is limited to a preset maximum threshold. This constraint aims to prevent the model from mechanically and meaninglessly piling up transition words or markers such as "however" and "wait" to infinitely obtain higher rewards, ensuring that the reasoning process generated by the model has substantial logical content rather than formal redundancy. Conversely, for output results whose accuracy meets the preset accuracy requirement, although the reward value decreases with the number of transition markers, it is limited to above a preset minimum threshold, ensuring that the model is not wrongly penalized for necessary complex reasoning. By combining an asymmetric reward strategy that dynamically adjusts accuracy with boundary constraints, this embodiment can effectively balance the simplicity and sufficiency of inference during training. It solves the problems of training instability and pseudo-deep inference caused by a single accuracy reward, and avoids the new trap of the model speculating by abusing over-labeling. This guides the multimodal large model to learn an efficient, realistic user experience diagnostic inference pattern with good generalization ability.

[0069] This embodiment does not limit the specific method of determining the reward value. For example, a preset reward function can be used to establish a mapping relationship between the reward value and the number of transition markers under different accuracy states. For instance, for output results whose accuracy meets the preset accuracy requirements, a monotonically decreasing linear function or an exponential decay function can be defined, such that as the number of transition markers increases, the reward value gradually decreases from a baseline positive value, and may even become negative, to strongly suppress redundant output when the answer is correct. For output results whose accuracy does not meet the preset accuracy requirements, a monotonically increasing logarithmic function or a saturated growth function can be defined, such that as the number of transition markers increases, the reward value gradually increases from a baseline negative value (i.e., the penalty is reduced), but the growth rate gradually slows down and approaches zero or a certain upper limit, to encourage in-depth thinking when the answer is incorrect while avoiding infinitely growing speculative behavior.

[0070] The following provides a specific calculation example based on piecewise functions.

[0071] In this embodiment, The final reward value, The number of transition markers, For accuracy indicators (when the prediction is correct) When the prediction is wrong For cases where accuracy requirements are met ( ), reward function It can be defined as a linearly decreasing function:

[0072] (1)

[0073] in, As the baseline positive reward value, It is a penalty coefficient greater than zero, used to control the reward points deducted for each additional transition marker.

[0074] A lower limit can also be set. This leads to the final reward .

[0075] For cases where accuracy requirements are not met ( ), reward function Defined as a saturated growth function, for example, in logarithmic form:

[0076] (2)

[0077] in, The initial negative reward value (baseline penalty). It is a growth coefficient greater than zero, used to adjust the degree of punishment reduction as the number of reasoning steps increases.

[0078] It also allows setting an upper limit. This leads to the final reward .

[0079] In the above formula, the parameters and The value of can be adjusted according to the distribution characteristics of the training data, for example... The possible values ​​are in the range [0.1, 1.0]. The possible values ​​are in the range [0.5, 2.0]. It is usually set to a positive value, such as 1.0. It is usually set to a negative value, such as -1.0. and The overall reward scale is set to ensure the stability of gradient updates. In addition to using explicit mathematical functions, this embodiment also supports determining the reward value by looking up a predefined reward lookup table, which stores fixed reward scores corresponding to different accuracy states and different intervals of transition marker quantity; or by using a mapping strategy based on piecewise linear interpolation to divide the number of transition markers into several intervals, and assigning different basic rewards and slope coefficients to each interval according to the accuracy state, thereby achieving more refined reward control.

[0080] In step 205, the parameters of the multimodal large model are adjusted using a reinforcement learning algorithm based on the reward value.

[0081] This embodiment does not limit the reinforcement learning algorithm used. For example, it can use Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), or other applicable reinforcement learning algorithms to use the reward value calculated in step 204 as a feedback signal to update the policy gradient of the multimodal large model.

[0082] For example, throughout the training process, the system uses the current multimodal large model (as a policy network) to reason about the first type of sample data, generating outputs containing thought chains and prediction conclusions. Subsequently, it calculates the Kullback-Leibler Divergence (KL Divergence) in conjunction with the Reference Model (usually an initialized frozen model) to constrain the magnitude of policy updates and prevent the model from deviating too far from the original language distribution, leading to content collapse or loss of language ability. Next, it constructs an Advantage Function based on the reward value and the KL Divergence penalty term, and calculates the gradient of the loss function through backpropagation to update the weight parameters of the multimodal large model.

[0083] Through this iterative training process, the model gradually learns an adaptive reasoning strategy: when accurately diagnosing user experience problems, it automatically optimizes its internal attention mechanism to generate a more concise thought chain, reducing ineffective transitional marking and redundant steps; while when faced with complex, ambiguous, or error-prone diagnostic scenarios leading to initial misjudgments, it is incentivized to activate deeper semantic analysis and visual feature extraction capabilities, generating detailed reasoning paths that include more reflection and correction steps. With increasing training rounds, the multimodal large model maintains or even improves the accuracy of user experience diagnosis while significantly reducing average inference length and computational latency, achieving a dual optimization of efficiency and accuracy. Ultimately, it yields a multimodal large model with efficient, accurate, and highly interpretable automated user experience diagnostic capabilities.

[0084] Generally, the quality of the sample data in the dataset is crucial to the training effect. The following describes the process of obtaining the first type of sample data in different embodiments.

[0085] When building datasets for training large multimodal models, increasing data size does not necessarily lead to improved model performance; instead, optimizing data quality is crucial. To improve the quality of sample data, in some implementations, the Perceptual Hash (pHash) algorithm can be used to deduplicate real user interface screenshots. The choice of pHash over embedding-based models (such as CLIP) for deduplication is primarily based on two considerations: First, the core goal of deduplication is to eliminate "visual redundancy," that is, to remove highly similar or identical duplicate screenshots at the pixel level, rather than identifying different interfaces with "semantic repetition." pHash can efficiently capture the visual fingerprint of an image, accurately matching visual duplicates. Second, there is a significant difference in computational cost between the two. Generating embedding vectors using deep learning models like CLIP requires GPU resources, with a single image processing time approaching 100ms; while the pHash algorithm only requires CPU resources, with a single image processing time of only about 9ms. Considering that the data cleaning process needs to be repeated monthly, and given the massive amounts of UI screenshot data, pHash offers significant advantages in computational efficiency and cost-effectiveness while maintaining high deduplication accuracy, making it more suitable for the preprocessing of large-scale industrial data. This approach can also introduce greater sample diversity, reducing overfitting to specific UI layouts.

[0086] In one implementation, the first type of sample data includes positive sample data and negative sample data. Positive sample data refers to sample data that does not conform to preset user experience rules, such as screenshots of interfaces with user experience defects; negative sample data refers to sample data that conforms to preset user experience rules, such as screenshots of normal interfaces without obvious user experience defects. Given that in real-world applications, the number of defective user interfaces (positive samples) obtained through data collection is usually far less than the number of defect-free interfaces (negative samples), and that user experience problems exhibit a long-tail distribution, if a traditional uniform sampling strategy is adopted, the training set will be filled with a large number of simple negative samples that contribute very little to the model's gradient updates (i.e., samples that the model easily judges as compliant). This imbalanced training not only wastes computational resources but may also cause the model to overly favor predicting "defect-free" interfaces during training, triggering reward hacking or gradient collapse in reinforcement learning, thereby reducing the model's ability to identify hidden and complex user experience problems.

[0087] To improve the quality of the dataset and address the aforementioned issues, this embodiment introduces a difficult negative sample mining mechanism. This mechanism dynamically filters the initial negative sample set, retaining high-value difficult negative samples and discarding low-value simple negative samples. Specifically, the process of obtaining negative sample data from the first type of sample data includes:

[0088] First, candidate negative samples are selected from the initial negative sample set. Second, the evaluation model (such as a pre-trained multimodal large model) is used to perform multiple independent sampling predictions on the candidate negative samples, resulting in multiple sampling results. The sampling results are used to characterize the evaluation model's classification results on whether the candidate negative samples conform to the preset user experience rules. Specifically, these sampling results can characterize the evaluation model's classification confidence or probability distribution on whether the candidate negative samples conform to the preset user experience rules.

[0089] Next, a consistency index is calculated for these multiple sampling results. This consistency index is used to quantify the degree of uncertainty or disagreement in the model's classification of the current candidate negative sample. For example, the consistency index can be the variance of multiple sampling results, entropy value, or the inverse of the proportion of the majority vote.

[0090] Finally, the calculated consistency index is compared with a preset threshold. If the consistency index is lower than the preset threshold, it indicates that the model's judgment of the sample is highly consistent across different samplings (i.e., the model considers the sample to be very simple and clearly a negative sample). Such samples are considered simple negative samples and are discarded or have their weights reduced. Conversely, if the consistency index is higher than or equal to the preset threshold, it indicates that the model's judgment of the sample has significant uncertainty or disagreement (i.e., the sample is near the decision boundary and is deceptive or hidden). Such samples are identified as difficult negative samples and included in the negative sample subset of the first type of sample data for subsequent training.

[0091] For example, when screening negative samples, eight independent sampling predictions are performed on each candidate negative sample image. The number of times the image is classified as "defect-free" (i.e., conforming to the preset user experience rules) in these eight sampling results is counted. If a candidate negative sample is classified as "defect-free" six or more times in eight runs, it indicates that the model's judgment of the sample is highly consistent, and it is identified as a simple negative sample. Since such samples are easy to distinguish, their contribution to the gradient update based on GRPO is minimal, so they are discarded from the training set. Conversely, if a candidate negative sample is classified as "defect-free" five or fewer times in eight runs, it indicates that the model's prediction of the sample is inconsistent or uncertain, and it is identified as a difficult negative sample.

[0092] By employing this uncertainty-based dynamic screening strategy, this embodiment can accurately locate high-value negative samples that are difficult for the model to distinguish and are prone to misjudgment. This forces the model to pay attention to subtle experience differences and complex boundary situations during training, thereby effectively suppressing reward hacking behavior and significantly improving the model's diagnostic accuracy and generalization ability for long-tail and hidden experience problems. At the same time, it improves the overall training efficiency by reducing the computational overhead of invalid samples.

[0093] In another implementation, to balance the number of positive and negative samples in the dataset and alleviate the class imbalance problem, this embodiment employs a data augmentation strategy to expand the positive sample set. Specifically, obtaining positive sample data from the first type of sample data includes: for each candidate positive sample (i.e., an interface screenshot with user experience defects) in the initial positive sample set, performing augmentation processing to generate one or more augmented candidate positive samples. The augmentation processing includes pixel-level augmentation and / or spatial-level augmentation. Pixel-level augmentation aims to simulate visual changes under different display environments, specifically including brightness adjustment (e.g., randomly adjusting the brightness of an image) and color jitter (e.g., randomly changing the saturation, contrast, or hue of an image); spatial-level augmentation aims to simulate the user's observation effect at different viewing angles or device resolutions, specifically including affine transformations (e.g., translation, rotation, scaling) and perspective transformations (e.g., simulating a tilted viewing angle).

[0094] To ensure that the enhanced samples retain the original user experience defect features and that the label validity is not affected, this embodiment strictly limits the perturbation amplitude of the enhancement process. For example, the brightness change is controlled to no more than 15%, the image scaling ratio is controlled to no more than 10%, and the rotation or cropping angle is controlled to no more than 5 degrees. This restricted enhancement strategy introduces sufficient diversity to prevent model overfitting while ensuring the semantic integrity of the enhanced image, so that the original positive sample labels (such as "pop-up window exists without a close button") still hold true in the enhanced image. Finally, the original candidate positive samples and the enhanced candidate positive samples are used together as positive sample data in the first class of sample data for model training. This approach effectively increases the proportion of positive samples in the training batch, improves the model's learning weights for minority class defect samples, and further improves the diagnostic performance of the multimodal large model in long-tailed distribution scenarios.

[0095] To mitigate the "catastrophic forgetting" problem that may occur when multimodal large models are post-trained for the user experience diagnosis vertical domain, this embodiment introduces a general domain data hybrid training mechanism. Catastrophic forgetting refers to the gradual degradation of the general visual-language understanding capabilities (such as basic OCR recognition, element localization, and semantic description) acquired during the pre-training stage when the model learns new knowledge in a specific domain (such as UI experience diagnosis) because the gradient update direction is too biased towards vertical domain features. To address this issue, this embodiment uses not only the first type of sample data (for experience diagnosis) during training but also acquires the second type of sample data and corresponding sample labels to construct a hybrid training dataset for cross-domain data regularization.

[0096] The second type of sample data can originate from large-scale general-purpose user interface corpora (such as the MultiUI dataset, which contains millions of UI samples from millions of websites, covering various multimodal tasks such as image description, web page question answering, OCR recognition, element localization, and action prediction). This data aims to maintain and enhance the basic user interface understanding capabilities of large multimodal models. The second type of sample data includes at least visual modal data of the user interface (such as screenshots), and may also include textual modal data and / or interaction data. The corresponding sample labels are used to characterize at least one of the semantic content, element location, or interactive actions of the user interface, for training semantic understanding, visual localization, and action prediction tasks, respectively. Specifically, the sample labels can be natural language text describing the overall or partial content of the interface (emphasizing semantic fidelity and linguistic fluency), bounding box coordinates of specific UI elements in the interface (for spatial localization), or prediction results of user action sequences.

[0097] During training, the second type of sample data is input into the multimodal large model to obtain the model's output prediction results. These prediction results are used to characterize the predictions of semantic content, element positions, or interactive actions within the user interface. Subsequently, a reward value is determined or a loss is calculated based on the difference between the prediction results and the sample labels. For example, for semantic understanding tasks, this can be evaluated by calculating the similarity or overlap between the predicted text and the labeled text; for visual localization tasks, it can be evaluated by calculating the intersection-over-union (IoU) ratio between the predicted bounding box and the ground truth bounding box.

[0098] Through multi-source data hybrid training, this embodiment achieves collaborative optimization of cross-domain capabilities. In each gradient update step, the model simultaneously receives gradient signals from both the UX diagnostic task and the general UI understanding task. The data and rewards from the general task act as implicit regularization constraints, limiting the overfitting of model parameters to a single UX diagnostic feature. This mechanism forces the model to retain a general perceptual foundation of UI element semantics, layout, and function while specializing in diagnosing experience defects. In other words, the model can only maximize the reward of the general task if it maintains a good general UI understanding capability; and it can only obtain the reward of the diagnostic task if it accurately diagnoses UX problems. The combination of these two aspects allows the model to find a balance point in the parameter space that is capable of both vertical domain diagnosis and general capabilities, thereby effectively avoiding catastrophic forgetting and achieving simultaneous improvement in specialized diagnostic capabilities and general basic capabilities.

[0099] Furthermore, the introduction of MultiUI not only ensures the model's basic general UI recognition capabilities and prevents catastrophic forgetting, but also further enhances the model's deep reasoning ability in user experience (UX) diagnosis. Specifically, the numerous fine-grained annotation tasks (such as element localization, OCR text recognition, and action prediction) included in the MultiUI dataset force the multimodal large model to establish a high-precision mapping relationship between visual features and semantic information. This accurate understanding of UI element attributes, spatial layout, and functional semantics constitutes a necessary prerequisite for complex UX reasoning. For example, only when the model can accurately identify the visual form of the "close button" and its relative position in the pop-up can it further infer the experience defect of "the pop-up has no close entry"; only by accurately understanding the page jump path and interaction logic can it determine whether the navigation process conforms to the principle of efficiency. Therefore, by training with mixed MultiUI data, the model not only learns to see interface elements, but also learns to understand the logical relationships between elements, thus providing a solid factual basis and logical support for structured UX diagnosis based on thought chains, significantly enhancing the model's reasoning depth and accuracy when dealing with hidden and long-tail experience problems.

[0100] In other embodiments, the gradient generated by the general understanding task can be used as a regularization term in each gradient update step to limit the drift of model parameters in the UX diagnosis direction. Specifically, the model first calculates the loss function for the UX diagnosis task. Loss function for general UI understanding tasks .in, Based on the asymmetric reward mechanism described above, the aim is to optimize the diagnostic accuracy and inference efficiency of the model. The loss function is determined based on standard supervised losses for general tasks (such as cross-entropy loss for semantic understanding and mean squared error loss for coordinate regression), aiming to maintain the model's basic perceptual capabilities. Subsequently, the system calculates these two loss functions relative to the model parameters. The gradient is denoted as and .

[0101] To implement regularization constraints at the gradient level, this embodiment employs a gradient projection method to correct the gradient of the UX diagnostic task. Specifically, the gradient of the general task is calculated. UX diagnostic task gradient Projection components in the direction, and from Subtracting the projected component from the middle yields the orthogonalized gradient. The mathematical expression is: .

[0102] This operation ensures that the parameter update direction of the UX diagnostic task remains orthogonal to or at an angle greater than 90 degrees to the direction of the gradient of the general task, thereby minimizing the negative impact on the performance of the general task while optimizing the performance of UX diagnostics. and If a conflict exists (i.e., the dot product is negative), then only retain [the product]. Zhongyu Non-conflicting components, or completely discarding update directions that lead to conflicts, are used to strictly limit the drift of model parameters in directions that impair generality.

[0103] Besides gradient projection, this embodiment can also use gradient weighted fusion to achieve regularization constraints. Specifically, the final update gradient... It consists of two parts: .in, This is a regularization coefficient, typically ranging from [0.1, 1.0], used to balance improving UX diagnostic capabilities with maintaining general-purpose capabilities. By adjusting... The magnitude of this value can dynamically control the pull of the gradient on parameter updates for general tasks. When When the parameters are large, the model parameters tend to remain near the optimal solution for the general task, effectively suppressing parameter divergence caused by UX-specific training; when When the model is smaller, it focuses more on optimizing the UX diagnostic task. This explicit constraint at the gradient level, compared to simple data mixing training, can more finely control the knowledge transfer and interference between different tasks, ensuring that while the model acquires specialized diagnostic capabilities, its underlying visual encoder and text encoder still have strong generalization and understanding capabilities, thus achieving a "specialized but not biased" training effect.

[0104] In current technologies, due to the heterogeneous output spaces and vastly different optimization objectives of different task types in user experience diagnostic tasks (focusing on logical reasoning and rule determination) and general UI understanding tasks (focusing on semantic recognition and spatial localization), joint optimization using a single reward metric often leads to gradient conflicts. This causes the model to rapidly lose its basic visual-language perception capabilities while improving its vertical domain diagnostic capabilities. Therefore, in one implementation, different reward functions can be dynamically matched to different tasks during reward calculation, based on the task type, to avoid inter-task interference and catastrophic forgetting problems caused by using a uniform reward function in traditional methods.

[0105] In this embodiment, when determining the reward value, the task type of the currently input second type of sample data can be identified first. The task type includes at least semantic understanding tasks, visual localization tasks, and action prediction tasks. This embodiment does not limit the specific identification method. For example, identification can be performed using task type identifiers. During the data preprocessing stage, each sample data (whether UX diagnostic data or general UI understanding data) can carry a task type identifier (TaskID). For example, Task ID=0 can be defined as a UX defect diagnosis task, Task ID=1 as a UI semantic description task, Task ID=2 as an element localization task, and Task ID=3 as an interaction action prediction task. Alternatively, automatic inference and identification can be performed based on the label format or data structure characteristics of the sample data. Specifically, if the label corresponding to a sample is a natural language text sequence, the sample is determined to belong to a semantic understanding task; if the label corresponding to a sample is a bounding box containing coordinate information or a set of polygon vertices, the sample is determined to belong to a visual localization task; if the label corresponding to a sample is a discrete action instruction code or a sequence of operation events (such as Click, Swipe, Type), the sample is determined to belong to an action prediction task. In addition, it can also be identified based on specific instruction keywords in the input prompts. For example, when the prompts contain instructions such as "describe the image content" or "extract text", it is classified as a semantic understanding task; when they contain instructions such as "detect element position" or "draw bounding box", it is classified as a visual localization task; and when they contain instructions such as "predict the next operation" or "generate interactive script", it is classified as an action prediction task.

[0106] Subsequently, based on the identified task type, a corresponding target reward function is dynamically matched from a pre-defined set of reward functions. Then, based on the difference between the prediction result and the sample label, the specific reward value is calculated using this target reward function. This task-aware reward routing mechanism aims to solve the reward conflict problem caused by heterogeneous targets in multi-task mixed training, achieving decoupling optimization between different tasks.

[0107] The semantic understanding task aims to enhance the model's ability to cognitively analyze the visual content of the user interface and its natural language generation capabilities. In this task, the model must generate natural language descriptions based on user interface screenshots or answer questions about the interface content. For example, the model may need to describe the overall layout of the interface (e.g., "the top is the navigation bar, and the middle is the product list") or explain the functional meaning of specific UI elements. For instance, the second type of sample data for the semantic understanding task can be user interface screenshots and optional text instructions (e.g., "Please describe this page"). The corresponding sample labels can be manually annotated standard natural language descriptions or standard answers. The model's prediction results for the second type of sample data can specifically be the predicted semantic content of the user interface, such as the natural language text sequence generated by the model. At the cross-domain regularization level, this task ensures that when the model performs UX diagnostic inference, its underlying visual semantic encoder will not lose its semantic understanding of normal UI elements due to excessive focus on violation features, thereby preventing the model from exhibiting cognitive biases that focus only on defects and fail to grasp the overall picture.

[0108] Visual localization tasks are used to train models to perceive the spatial location of specific elements in a user interface with high accuracy. In this task, the model needs to accurately pinpoint the location of the corresponding UI element in a screenshot or screen recording based on textual references (such as "the close button in the upper right corner"). These tasks focus on establishing fine-grained visual-spatial mappings, which are fundamental for identifying user experience issues such as occlusion and layout rationality. For example, the second type of sample data for visual localization tasks can be user interface screenshots and textual instructions describing the target element. The corresponding sample labels can be ground truth bounding boxes, and the model's prediction of the second type of sample data can be the predicted element position, typically expressed as bounding box coordinates, including the coordinates of the upper left and lower right corners. Or center point and width / height. In preventing catastrophic forgetting, visual localization is fundamental in UX diagnostics for identifying spatial defects such as "occlusion" and "layout clutter." By continuously optimizing the reward for this task, the model can ensure that its visual backbone network's sensitivity to pixel-level spatial relationships does not become dulled as it learns UX rules, thus guaranteeing the accuracy of defect location coordinates in the diagnostic results.

[0109] Action prediction tasks are used to train the model's depth of understanding of user interaction logic and business processes. In this task, the model needs to predict the next interactive action to be performed based on the current interface state and user intent, or plan the complete sequence of operations required to complete a task. This type of task focuses on understanding the causal logic between interface state, user intent, and interactive behavior, which helps to evaluate the efficiency and usability of the operation process. For example, the second type of sample data for the action prediction task can be user interface screenshots and user intent commands (such as "login account" or "buy a red T-shirt"). The corresponding sample labels can be manually labeled standard interactive action sequences. The model's prediction result for the second type of sample data can be a predicted interactive action sequence, including action types (such as Click, Swipe, Type input) and the corresponding target element or coordinates. In cross-domain regularization, this task ensures that the model still possesses correct interaction common sense when diagnosing logical experience problems such as navigation redundancy and broken operation paths, avoiding misjudgments of normal interaction processes due to overfitting of vertical domain rules.

[0110] Specifically, when the task type is a semantic understanding task (e.g., generating natural language descriptions of user interfaces or explaining the functionality of UI elements), the objective reward function is configured as an evaluation metric based on text similarity. The system calculates the similarity between the predicted semantic content in the prediction result and the reference semantic content in the sample labels, for example, using the ROUGE-L metric to measure the longest common subsequence between the generated text and the reference description. A high similarity results in a high reward; conversely, a low similarity results in a lower reward. This metric can evaluate information coverage and text coherence while allowing for reasonable wording variations, thus more accurately reflecting the quality of open-ended text generation tasks and ensuring that the model achieves semantic fidelity while maintaining linguistic fluency.

[0111] When the task type is a visual localization task (e.g., predicting the spatial position of UI elements on the screen), the objective reward function is configured as an evaluation metric based on position matching. The system obtains the predicted element position (usually represented as bounding box coordinates) from the prediction results and the ground truth element position from the sample labels, and calculates the position matching degree between the two. In one example, a center point matching strategy is used, which determines whether the center point of the predicted bounding box falls within the interior region of the ground truth bounding box. If the center point is within the ground truth bounding box, localization is considered successful, and a positive reward is given; otherwise, localization is considered unsuccessful, and a negative reward or zero reward is given. This binarized or continuous position reward can accurately guide the model to learn the spatial layout features of UI elements.

[0112] When the task type is an action prediction task (e.g., predicting the user's next interaction on the interface), the target reward function is configured as an evaluation metric based on behavior matching. The system compares the predicted interaction action sequence in the prediction results with the actual interaction action sequence in the sample labels, calculating the consistency between the two in terms of action type, order, and parameters. If the predicted action completely matches the actual action or is consistent within the tolerance error range, a high reward is given; otherwise, the reward value is reduced.

[0113] Furthermore, for the UX issue detection task involved in the first type of sample data (such as the multiple-choice question format mentioned above), its objective reward function is independent of the aforementioned general task and is configured as a binary judgment logic based on answer accuracy. That is, a positive reward is only given when the predicted structured option (such as A, B, C) is completely consistent with the standard answer. Through this dynamic routing mechanism, this embodiment can apply a better verifiable reward function (Reinforcement Learning with Verifiable Rewards, RLVR) for the characteristics of each task, avoiding inter-task interference and gradient conflicts caused by using a uniform reward function in traditional methods. This not only effectively prevents the model from catastrophic forgetting when learning vertical domain UX diagnostic knowledge, but also ensures the continuous evolution of the model's general UI understanding ability, achieving a synergistic improvement in specialized diagnostic capabilities and general basic capabilities.

[0114] In one implementation, since the original reward values ​​output by different sub-reward calculators have different numerical distributions and dimensions (for example, the location reward based on IOU ranges from [0, 1], while the UX diagnostic reward based on the human preference model may range from [-5, 5]), direct addition can lead to an imbalance in gradient dominance. Therefore, in this embodiment, dynamic standardization can be performed on the original reward values ​​before route merging.

[0115] For example, a sliding window statistician can be maintained to record the average reward of each task over the most recent N training steps in real time. and standard deviation For the first The original reward generated by each task Converted into standardized rewards using the Z-Score standardized formula : (3) Among them, To prevent small constants with zero denominators, this process maps the reward values ​​of all tasks to a standard normal distribution with a mean of 0 and a variance of 1, eliminating dimensional differences and making the gradients of different tasks comparable during backpropagation.

[0116] In one implementation, to overcome the problems of general multimodal large models lacking fine-grained expertise in user experience diagnosis and being unable to accurately identify specific regulatory violations (such as being able to identify "a pop-up exists in the middle of the page" but lacking knowledge related to user experience such as "pop-ups must provide a clear close button, pop-ups must not appear continuously to interfere with the user, and pop-ups must not obscure key information"), and the potential training convergence problem that pure reinforcement learning algorithms may cause, this embodiment adopts a two-stage training strategy. Figure 3 As shown, the first stage is Supervised Fine-Tuning (SFT), which aims to inject pre-defined user experience rule knowledge into the model through the mental chain labels of the reasoning process; the second stage is reinforcement learning, which is the training process in the above embodiment, and aims to optimize the model's reasoning efficiency and decision accuracy.

[0117] Specifically, before inputting the first type of sample data into the multimodal large model for reinforcement learning, this embodiment also utilizes a supervised fine-tuning dataset to perform supervised fine-tuning training on the multimodal large model. The supervised fine-tuning dataset includes the first type of sample data (e.g., multimodal data containing UI screenshots), corresponding sample labels (e.g., compliance / violation classification or multiple-choice options), and associated inference paths. The inference path is a structured logical deduction process built based on preset user experience rules, used as a thought chain label, and its content can include the following three key steps:

[0118] Step 1, Perception and Recognition: Identify whether there are target elements or visual features in the user interface that may violate the preset user experience rules (e.g., "A modal pop-up window covering approximately 40% of the page was detected in the center").

[0119] Step 2, Analysis and Attribution: Combining visual information and interaction logic, determine the specific reasons or missing elements that led to the rule violation (e.g., "Inspecting the top and bottom areas of the pop-up, no explicit 'close' icon or text button was found, and no interactive prompt that could be closed by clicking the overlay was detected").

[0120] Step 3, Rule Mapping and Conclusion: Refer to the specific clauses of the preset user experience rules on which it is based, and draw the final diagnostic conclusion (e.g., "According to Article 3.2 of the Mobile Pop-up Design Specification, 'All non-mandatory, highly intrusive pop-ups must provide a clear and easy-to-click close entry,' it is determined that the interface has the experience defect of 'pop-ups without a close button'").

[0121] Through this supervised fine-tuning that includes a complete logical chain, the model not only learns the final classification result but also internalizes the complete reasoning paradigm from visual perception to rule determination. This enables the model to establish semantic associations between visual features (such as missing button icons) and abstract rules (such as "a close entry must be provided"), thereby possessing the cognitive ability to understand fine-grained experience specifications.

[0122] After supervised fine-tuning, the model enters the second stage of reinforcement learning training. At this point, the model possesses basic domain knowledge and reasoning capabilities, enabling it to generate logical thought chains. Building upon this, the asymmetric reward mechanism described in the previous embodiment (i.e., determining the reward value based on accuracy and the number of transition labels) is applied to further optimize the model's reasoning process. Since the correct rule knowledge has already been injected in the SFT stage, the focus of the RL stage is no longer on teaching the model to "learn" the rules, but rather on teaching the model "how to efficiently apply the rules"—that is, simplifying redundant transition labels while ensuring correct reasoning, and maintaining sufficient reasoning depth when facing complex and challenging samples. This two-stage training mechanism of "SFT knowledge injection + RL efficiency optimization" effectively solves the dual problems of insufficient domain knowledge and inefficient reasoning processes in general models, significantly improving the professionalism and practicality of multimodal large models in automated user experience diagnosis scenarios. Furthermore, no further annotation of thought chains is required in the reinforcement learning stage, reducing the workload of annotation.

[0123] This embodiment does not limit the specific method for obtaining the reasoning path (i.e., thought chain label) associated with the first type of sample data, and various approaches can be used to construct supervised fine-tuning data. For example, domain experts can manually annotate the data based on preset user experience rules to ensure the rigor and standardization of the reasoning logic; alternatively, an automated or semi-automated approach can be used, where preset user experience rules, the first type of sample data, and the corresponding sample labels are input into a pre-trained teacher model, and the reasoning process generated by the teacher model is used as the thought chain label; or, the first type of sample data can be input into the teacher model, and only when the prediction result output by the teacher model is consistent with the standard sample label is the output reasoning process selected and adopted as the thought chain label, thereby ensuring the logical correctness of the thought chain label through result consistency verification.

[0124] For example, the content of a mind chain tag can be as follows:

[0125] Step 1 [Perception]: There is a modal pop-up window in the center of the page that occupies 40% of the screen, with a semi-transparent overlay on the background.

[0126] Step 2 [Analysis]: The pop-up title reads "Open a membership and enjoy a 50% discount," and the content is marketing information. A careful scan of the pop-up's perimeter and corners revealed no "X" close icon or "cancel" button; furthermore, simulating clicking the covered area yielded no feedback indicating it could be closed.

[0127] Step 3 [Reasoning]: According to the "Pop-up Interference Control Specifications" in the preset user experience rule base, non-mandatory marketing pop-ups must provide users with a clear and easily discoverable way to close them, in order to protect users' right to choose and their sense of control. The current interface lacks this necessary interactive element.

[0128] Step 4 [Conclusion]: The existence of "pop-up window without a close button" violates the user experience rules and constitutes a defect that seriously interferes with user operation.

[0129] During supervised fine-tuning, a composite loss function is constructed to update parameters by calculating the differences between the model's output reasoning process and the thought chain labels, as well as the differences between the predicted conclusions and the sample labels. Specifically, this loss function typically includes two parts: first, the inference path loss, which measures the difference between the intermediate thought chain steps generated by the model (such as perception, analysis, rule referencing, etc.) and the labeled thought chain labels in the text sequence. This can be calculated using cross-entropy loss and aims to guide the model to learn a logical deduction paradigm that conforms to preset user experience rules; second, the predicted conclusion loss, which measures the difference between the model's final output diagnostic results (such as violation type classification or compliance judgment) and the real sample labels. This can also be calculated using cross-entropy loss and aims to ensure the accuracy of the model's diagnostic conclusions.

[0130] Subsequently, based on the composite loss value, the parameters of the multimodal large model are updated using the backpropagation algorithm. Through this joint optimization mechanism, the model not only approximates the true label in the final output, but also aligns its internal reasoning logic with the high-quality thought chain of expert knowledge or teacher models. This enables the model to gradually internalize the complete cognitive chain from visual feature recognition to rule mapping, thus possessing a solid domain knowledge foundation in subsequent reinforcement learning stages. It can effectively distinguish subtle experiential differences (such as "there is a close button but its location is hidden" versus "there is no close button"), avoiding illusions or misjudgments caused by a lack of rule understanding.

[0131] This specification also provides an embodiment of a method for user experience diagnosis based on a multimodal large model, wherein the multimodal large model is trained based on any of the methods described in the above embodiments, such as... Figure 4 As shown, the method may include the following steps:

[0132] In step 401, user operation data is obtained.

[0133] This embodiment does not limit the source of user operation data, which is obtained with the user's authorization or permission. For example, it can be real-time interaction logs generated by real users during the use of terminal devices (such as smartphones, tablets, personal computers, etc.), test data generated by simulating user behavior through automated test scripts, or existing user session records extracted from historical databases.

[0134] User interaction data includes at least visual modal data of the user interface, and may also include textual modal data and / or interaction data of the user interface. The definitions of the above data are provided above and will not be repeated here. For example, the user interface can be as follows: Figure 1 The user screenshots shown fully preserve the interface's layout, colors, icons, and visual hierarchy information; text modal data can be... Figure 1 The text on the page and buttons can be extracted from the screenshot using OCR technology, or it can be obtained directly from the application's underlying text resources; the interaction data can be the sequence of operations performed by the user on the interface, including but not limited to click, swipe, long press, input and other action events, as well as the coordinate position, timestamp and trigger object identifier when these actions occur.

[0135] In step 402, user operation data is input into the multimodal large model, which evaluates the user experience of the user interface based on preset user experience rules to obtain diagnostic results.

[0136] In this step, the multimodal large model utilizes the domain knowledge (i.e., pre-defined user experience rules) learned during training or invokes pre-defined user experience rules and reasoning capabilities from the knowledge base to comprehensively analyze user operation data. For example, the model first processes visual modal data through a visual encoder to extract the spatial layout, visual style, and image features of interface elements; simultaneously, it processes text modal data through a text encoder to understand interface semantics; if interaction data exists, the model also analyzes the user's operation path and intent based on the interaction sequence. Subsequently, the model performs logical reasoning based on its internally constructed thought chain to generate user experience diagnostic results. For example, the thought chain may include the four reasoning processes mentioned in the above embodiments: perception, analysis, reasoning, and memory conclusions.

[0137] The diagnostic results indicate whether the user interface conforms to preset user experience rules. Diagnostic results typically include: a diagnostic conclusion, indicating whether there are experience defects in the current interface or interaction flow; if so, the specific defect type (e.g., "pop-up without a close button," "obstruction of key operation entry points," "insufficient color contrast," "redundant navigation path," etc.); the location of the problem; if a defect is detected, the model can also output the specific coordinates of the defect area or the identifier of the relevant UI element; and remediation suggestions. Based on the diagnosed defect type and the user experience specifications, the model generates targeted optimization suggestions to assist developers in rectification. For example, remediation suggestions can cite specific design guidelines, such as, "According to the 'Mobile Pop-up Design Specifications,' non-mandatory pop-ups must provide a close entry point in a prominent position." Remediation suggestions can also provide specific modification solutions to offer actionable operational suggestions, such as adding an 'X' close icon in the upper right corner of the pop-up, with a size no smaller than 44x44pt, and ensuring the clickable area is large enough; or changing the text color from #999999 to #666666 to improve the contrast with the white background to above 4.5:1.

[0138] In some embodiments, prompts can be input into a multimodal large model along with user action data to guide the model's input of diagnostic results, problem location, and repair suggestions.

[0139] For example, for Figure 1 In the user interface, one type of input prompt can be:

[0140] Based on the provided screenshot of the single user interface, diagnose whether there are any issues that violate user experience guidelines. Pay particular attention to the pop-up's interaction logic, visual hierarchy, and user control.

[0141] Analysis steps required:

[0142] Perception: Identify the main elements in the interface (pop-ups, text, buttons, close buttons, etc.).

[0143] Analysis: Based on the perceived information, examine the status of key interactive elements. For example, check whether the pop-up provides a clear and easy-to-use way to close it; check whether the visual weight of the main operation button and the secondary operation (closing) is balanced, and whether there are any risks of misleading design.

[0144] Reasoning: Compare the analysis results with the preset user experience rules to determine whether the current design meets the requirements of the terms. If there are deviations, analyze their severity.

[0145] Conclusion: Provide the final diagnostic results, determine whether it conforms to the specifications, and if not, indicate the specific defect type and location, and provide specific improvement solutions.

[0146] For example, regarding Figure 1 In the user interface, another input prompt could be:

[0147] Based on the provided screenshot of a single user interface, diagnose whether there are any issues that violate user experience guidelines. If so, indicate the specific type and location of the issue, and provide concrete improvement solutions.

[0148] This embodiment, through the multimodal large model obtained by the above training method, can adaptively adjust the inference process according to the difficulty of the user experience diagnosis task during the application phase, thereby maintaining a high diagnostic accuracy while reducing computational resource consumption. Specifically, since the model learns an accuracy-based asymmetric length reward mechanism during training, when the model faces a simple user interface scenario and can make a correct judgment, it tends to generate short inference paths, reducing meaningless transitional markers and repetitive descriptions. This directly compresses the length of the model's output text, reduces the computational load and response latency required for a single inference, and meets the efficiency requirements of large-scale real-time detection. Conversely, when the model faces complex, hidden, or error-prone user experience problems, if the initial judgment has uncertainty or error risk, the model will be incentivized to expand the inference steps, performing more detailed visual element perception, interaction logic analysis, and rule comparison. This dynamically increased inference depth helps the model correct potential biases and capture subtle experience defects, thereby ensuring that the diagnostic accuracy in difficult scenarios does not decrease due to the pursuit of speed.

[0149] Furthermore, the model leverages structured thought chain knowledge injected during the supervised fine-tuning phase, resulting in highly interpretable and standardized diagnostic results in the application phase. Instead of simply outputting a compliance or violation label, the model clearly identifies specific elements and their locations in the existing interface that violate preset user experience rules, following a logical sequence of perception, analysis, reasoning, and conclusion. It then generates specific remedial suggestions based on relevant design principles. This structured output not only facilitates developers' rapid location of problematic areas in the code but also avoids the illusion phenomenon common in general-purpose large models—preventing the model from fabricating non-existent interface elements or referencing incorrect specification clauses. Therefore, the method in this embodiment effectively solves the problems of traditional automated detection tools' inability to understand semantics and the redundancy and lack of domain expertise in the reasoning of existing large models. It achieves end-to-end automated diagnosis from visual perception to rule determination, significantly improving the efficiency and coverage of user experience governance.

[0150] Figure 5 This is a schematic diagram of the training device for a multimodal large model used for user experience diagnosis, as described in the embodiments of this specification. This device can be applied to any device, platform, or device cluster with computing and processing capabilities. The device includes:

[0151] The acquisition module 51 is used to acquire the first type of sample data and the corresponding sample labels. The first type of sample data includes at least the visual modal data of the user interface, and the sample labels are used to indicate the evaluation results of the user interface based on preset user experience rules.

[0152] The reasoning module 52 is used to input the first type of sample data into the multimodal large model and obtain the output results of the multimodal large model. The output results include the reasoning process and the prediction conclusion. The reasoning process is a thought chain that includes multiple intermediate reasoning steps. The reasoning process includes several transition markers, which are used to indicate the turning point of the reasoning process. The prediction conclusion is used to characterize the prediction evaluation result of the user interface based on the preset user experience rules.

[0153] The judgment module 53 is used to determine the accuracy of the output result based on the difference between the prediction conclusion and the sample label;

[0154] The reward module 54 is used to determine the reward value based on the accuracy of the output result and the number of transition markers during the inference process. When determining the reward value, for output results whose accuracy meets the preset accuracy requirements, the reward value decreases as the number of transition markers increases; for output results whose accuracy does not meet the preset accuracy requirements, the reward value increases as the number of transition markers increases.

[0155] The parameter tuning module 55 is used to adjust the parameters of a multimodal large model based on the reward value using a reinforcement learning algorithm.

[0156] In one implementation, the first type of sample data includes positive sample data and negative sample data. Positive sample data refers to sample data that does not conform to the preset user experience rules, while negative sample data refers to sample data that conforms to the preset user experience rules. The acquisition module 51 is specifically used to perform multiple sampling predictions using the evaluation model for each candidate negative sample in the initial negative sample set, obtaining multiple sampling results. The sampling results are used to characterize the classification result of the evaluation model on whether the candidate negative sample conforms to the preset user experience rules. The consistency index of the multiple sampling results is calculated. The consistency index is used to characterize the uncertainty of the classification result of the evaluation model on the candidate negative sample. The consistency index is compared with a preset threshold. If the consistency index is lower than the preset threshold, the candidate negative sample is determined to be negative sample data in the first type of sample data.

[0157] In one implementation, the first type of sample data includes positive sample data and negative sample data. The positive sample data is sample data that does not conform to the preset user experience rules, and the negative sample data is sample data that conforms to the preset user experience rules. The acquisition module 51 is specifically used to perform enhancement processing on each candidate positive sample in the initial positive sample set to obtain enhanced candidate positive samples. The enhancement processing includes pixel-level enhancement and / or spatial-level enhancement. Pixel-level enhancement includes brightness adjustment and color dithering, and spatial-level enhancement includes affine transformation and perspective transformation. The candidate positive samples and the enhanced candidate positive samples are used as positive sample data in the first type of sample data.

[0158] In one implementation, the acquisition module 51 is further configured to acquire a second type of sample data and corresponding sample labels. The second type of sample data is used to train the user interface understanding ability of the multimodal large model. The second type of sample data includes at least the visual modal data of the user interface, and the corresponding sample labels are used to characterize at least one of the semantic content, element position, or interactive action of the user interface. The inference module 52 is further configured to input the second type of sample data into the multimodal large model to obtain the prediction result output by the multimodal large model. The prediction result is used to characterize at least one of the predicted semantic content, predicted element position, or predicted interactive action of the user interface. The reward module 54 is further configured to determine a reward value based on the difference between the prediction result and the sample labels.

[0159] In one implementation, the reward module 54 is specifically used to identify the task type to which the currently input second type of sample data belongs. The task type includes at least one of the following: semantic understanding task, visual localization task, and action prediction task. Based on the task type, a corresponding target reward function is matched from a preset set of reward functions, and a reward value is calculated using the target reward function based on the difference between the prediction result and the sample label. Specifically, when the task type is a semantic understanding task, the target reward function is configured to calculate the reward value based on the text similarity between the predicted semantic content in the prediction result and the semantic content in the sample label; when the task type is a visual localization task, the target reward function is configured to calculate the reward value based on the positional matching degree between the predicted element position in the prediction result and the element position in the sample label; when the task type is an action prediction task, the target reward function is configured to calculate the reward value based on the behavioral matching degree between the predicted interactive action in the prediction result and the interactive action in the sample label.

[0160] In one implementation, the parameter tuning module 55 is further configured to perform supervised fine-tuning training on the multimodal large model using a supervised fine-tuning dataset before inputting the first type of sample data into the multimodal large model and obtaining the output result of the multimodal large model; the supervised fine-tuning dataset includes the first type of sample data, the corresponding sample labels, and the associated inference path; wherein, the inference path includes a logical deduction process based on preset user experience rules, and the logical deduction process includes at least: identifying whether there are target elements in the user interface that violate the preset user experience rules, determining the violation reasons that cause the violation, and referencing the terms of the preset user experience rules on which it is based.

[0161] In one implementation, the first type of sample data further includes text modal data and / or interaction data of the user interface, wherein the interaction data includes a sequence of user actions on the user interface, and the sequence of actions includes at least one of click location, page jump path and dwell time.

[0162] This specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method described in any of the above embodiments.

[0163] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the above embodiments.

[0164] This specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0165] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0166] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0168] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0169] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A training method for a multimodal large model for user experience diagnosis, the method comprising: Obtain a first type of sample data and corresponding sample labels. The first type of sample data includes at least visual modal data of the user interface. The sample labels are used to indicate the evaluation results of the user interface based on preset user experience rules. The first type of sample data is input into the multimodal large model to obtain the output result of the multimodal large model. The output result includes the reasoning process and the prediction conclusion. The reasoning process is a thought chain that includes multiple intermediate reasoning steps. The reasoning process includes several transition markers. The transition markers are used to indicate the turning point of the reasoning process. The prediction conclusion is used to characterize the prediction evaluation result of the user interface based on the preset user experience rules. The accuracy of the output result is determined based on the difference between the prediction conclusion and the sample label; The reward value is determined based on the accuracy of the output result and the number of transition markers during the inference process. When determining the reward value, for output results whose accuracy meets the preset accuracy requirement, the reward value decreases as the number of transition markers increases; for output results whose accuracy does not meet the preset accuracy requirement, the reward value increases as the number of transition markers increases. Based on the reward value, the parameters of the multimodal large model are adjusted using a reinforcement learning algorithm.

2. The method according to claim 1, wherein, The first type of sample data includes positive sample data and negative sample data. The positive sample data is sample data that does not conform to the preset user experience rules, and the negative sample data is sample data that conforms to the preset user experience rules. The step of obtaining negative sample data from the first type of sample data includes: For each candidate negative sample in the initial negative sample set, the evaluation model is used to perform multiple sampling predictions to obtain multiple sampling results. The sampling results are used to characterize whether the evaluation model classifies the candidate negative sample according to the preset user experience rules. Calculate a consistency index for the multiple sampling results, the consistency index being used to characterize the degree of uncertainty of the evaluation model's classification result for the candidate negative samples; The consistency index is compared with a preset threshold. If the consistency index is lower than the preset threshold, the candidate negative sample is determined to be negative sample data in the first type of sample data.

3. The method according to claim 1, wherein, The first type of sample data includes positive sample data and negative sample data. The positive sample data is sample data that does not conform to the preset user experience rules, and the negative sample data is sample data that conforms to the preset user experience rules. The step of obtaining positive sample data from the first type of sample data includes: For each candidate positive sample in the initial positive sample set, enhancement processing is performed to obtain enhanced candidate positive samples; The enhancement process includes pixel-level enhancement and / or spatial-level enhancement. The pixel-level enhancement includes brightness adjustment and color dithering, and the spatial-level enhancement includes affine transformation and perspective transformation. The candidate positive samples and the enhanced candidate positive samples are used as positive sample data in the first type of sample data.

4. The method according to claim 1, wherein, The method further includes: acquiring a second type of sample data and corresponding sample labels, wherein the second type of sample data is used to train the user interface understanding ability of the multimodal large model, and the second type of sample data includes at least visual modal data of the user interface, and the corresponding sample labels are used to characterize at least one of the semantic content, element position or interactive action of the user interface. The second type of sample data is input into the multimodal large model to obtain the prediction result output by the multimodal large model. The prediction result is used to characterize at least one of the predicted semantic content, predicted element position, or predicted interaction action of the user interface. The reward value is determined based on the difference between the prediction result and the sample label.

5. The method according to claim 4, wherein, The step of determining the reward value based on the difference between the prediction result and the sample label includes: Identify the task type to which the currently input second type of sample data belongs, wherein the task type includes at least one of the following: semantic understanding task, visual localization task, action prediction task; According to the task type, a corresponding target reward function is matched from a preset set of reward functions, and the reward value is calculated based on the difference between the prediction result and the sample label using the target reward function; Wherein, when the task type is the semantic understanding task, the target reward function is configured to calculate the reward value based on the text similarity between the predicted semantic content in the prediction result and the semantic content in the sample label; When the task type is the visual positioning task, the target reward function is configured to calculate the reward value based on the position matching degree between the predicted element position in the prediction result and the element position in the sample label. When the task type is the action prediction task, the target reward function is configured to calculate the reward value based on the behavioral matching degree between the predicted interactive action in the prediction result and the interactive action in the sample label.

6. The method according to claim 1, wherein, Before inputting the first type of sample data into the multimodal large model and obtaining the output result of the multimodal large model, the method further includes: performing supervised fine-tuning training on the multimodal large model using a supervised fine-tuning dataset; The supervised fine-tuning dataset includes the first type of sample data, the corresponding sample labels, and the associated inference paths; The reasoning path includes a logical deduction process based on the preset user experience rules. The logical deduction process includes at least: identifying whether there are target elements in the user interface that violate the preset user experience rules, determining the reasons for the violation, and referencing the terms of the preset user experience rules on which it is based.

7. The method according to claim 1, wherein, The first type of sample data also includes text modal data and / or interaction data of the user interface, wherein the interaction data includes a sequence of user operations on the user interface, and the sequence of operations includes at least one of click location, page jump path and dwell time.

8. A method for user experience diagnosis based on a multimodal large model, wherein, The multimodal large model is trained based on any one of the methods in claims 1-7, the method comprising: Acquire user operation data, which includes at least visual modal data of the user interface; The user operation data is input into the multimodal big model, which evaluates the user experience of the user interface based on preset user experience rules to obtain a diagnostic result. The diagnostic result is used to indicate whether the user interface conforms to the preset user experience rules.

9. A training device for a multimodal large model for user experience diagnosis, the device comprising: The acquisition module is used to acquire a first type of sample data and corresponding sample labels. The first type of sample data includes at least visual modal data of the user interface, and the sample labels are used to indicate the evaluation result of the user interface based on preset user experience rules. The inference module is used to input the first type of sample data into the multimodal large model and obtain the output result of the multimodal large model. The output result includes the inference process and the prediction conclusion. The inference process is a thought chain including multiple intermediate inference steps. The inference process includes several transition markers. The transition markers are used to indicate the turning point of the inference process. The prediction conclusion is used to characterize the prediction evaluation result of the user interface based on the preset user experience rules. The judgment module is used to determine the accuracy of the output result based on the difference between the prediction conclusion and the sample label; The reward module is used to determine a reward value based on the accuracy of the output result and the number of transition markers during the inference process. When determining the reward value, for output results whose accuracy meets the preset accuracy requirement, the reward value decreases as the number of transition markers increases; for output results whose accuracy does not meet the preset accuracy requirement, the reward value increases as the number of transition markers increases. The parameter tuning module is used to adjust the parameters of the multimodal large model based on the reward value using a reinforcement learning algorithm.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-7 or 8.

Images