Multi-model evaluation method and system based on scale self-calibration and phased decision

Abstract

The application is a multi-model evaluation method and system based on scale self-calibration and phased decision, which divides the evaluation process into multiple logically progressive stages: 1) single model structured annotation, independent execution of main problem identification and high-quality reason identification for each model to be evaluated; 2) rule mapping, mapping the annotation results of 1) into a pure numerical fact archive; 3) double model comparison evaluation, only based on the mapping results of 2) to perform objective numerical difference calculation; 4) multi-dimensional decision reasoning, only based on the difference comparison results of the third stage to make a final judgment. There is a strict data dependency relationship between each stage, the structured output of the previous stage can only be accessed, and the original model output text cannot be directly accessed, thereby realizing the complete decoupling of fact extraction and decision reasoning at the system architecture level. The application can realize dynamic self-calibration of the evaluation scale, construct an interpretable phased decision-making process, and effectively reuse artificial annotation experience.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a multi-model evaluation method and system based on scale self-calibration and phased adjudication. Background Technology

[0002] With the large-scale application of Large Language Models (LLMs) in scenarios such as intelligent question answering, content generation, and dialogue interaction, performance comparison and evaluation among multiple models has become a key step for enterprises to optimize products and enhance their core competitiveness. Currently, the mainstream large model comparison and evaluation methods in the industry are mainly divided into three categories: manual annotation methods, fixed-rule automatic evaluation methods, and single-step large model automatic adjudication methods.

[0003] Manual annotation relies on domain experts to judge the quality of model output. While it can ensure a certain level of accuracy, it has inherent drawbacks such as high cost, low efficiency, and strong subjectivity. Different annotators have naturally different judgment criteria, making it difficult to form a unified and stable evaluation standard. Fixed rule automatic evaluation method automates the evaluation process by quantitatively scoring through a preset indicator system. However, the rules are directly written by experts, relying on subjective experience. The rule content is out of touch with the actual annotated data, the rules are rigid and cannot adapt to changes in business scenarios and data distribution shifts. Furthermore, it does not consider the advantages and fatal flaws of the model output, which can easily lead to the problem of evaluation standards being out of touch with actual needs. Single-step large model automatic adjudication method directly calls the output of the large model to "win / draw / loss" conclusion. It is easy to operate, but the evaluation process is like a "black box". It lacks intermediate reasoning basis, cannot trace the logic of result generation, and does not integrate historical annotation experience to form a structured knowledge base, resulting in poor evaluation consistency.

[0004] In practical engineering applications, the above methods generally expose four core problems: First, the evaluation criteria lack dynamic calibration capabilities. Evaluation standards are prone to drift in different batches and scenarios. Models of the same quality may produce significantly different judgment results, resulting in a lack of comparability of long-term evaluation data. Second, the evaluation process lacks a logical hierarchical design, fails to build a progressive thinking chain structure, does not integrate a knowledge base of winning / losing modes, and couples factual analysis with adjudication reasoning, making it impossible to pinpoint the root cause of problems when results are abnormal. Third, the adjudication process lacks structured factual support, relying directly on the original output for judgment, which is easily influenced by irrelevant factors such as expression style and the elegance of language, resulting in insufficient objectivity and interpretability of the results. Fourth, human experience has low reusability. The implicit experience of annotators in identifying core strengths and major problems is difficult to transform into explicit rules. New annotators or models cannot be effectively reused, causing the evaluation criteria to fluctuate with personnel changes.

[0005] These shortcomings severely restrict the engineering implementation and large-scale application of large-scale model comparison and evaluation, and there is an urgent need for a new evaluation method that combines high consistency, high interpretability and high reusability. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the technical problem to be solved by this invention is to provide a multi-model evaluation method and system based on scale self-calibration and phased adjudication, which is used for comparative evaluation of multi-model outputs, can realize dynamic self-calibration of evaluation scale, construct an interpretable phased adjudication process, and effectively reuse manual annotation experience.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: Firstly, the present invention provides a multi-model evaluation method based on scale self-calibration and phased adjudication, the method comprising the following rigorously progressive and data-decoupled steps: Step S1, Evaluation Scale Self-calibration: Based on historical manually calibrated data, the scale features and pattern features are automatically extracted through a large language model combined with a rule-based reasoning algorithm to construct a dual knowledge base and evaluation scale constraint information; the dual knowledge base includes a victory mode knowledge base and a failure mode knowledge base; the evaluation scale constraint information includes scenario-based tolerance thresholds, boundary judgment criteria, and the mapping relationship between the severity level of major problems and the advantage level of excellent reasons to the pass rate level; Step S2, Single-model structured annotation: For multiple large models to be evaluated, the evaluation scale constraint information and dual knowledge bases are loaded respectively. The large language model performs fact extraction on the original answers of each model and outputs only the structured annotation results. The fact extraction includes two aspects: main problem identification and high-quality reason identification. Main problem identification is used to identify and label the main problems from the preset label system, prioritizing the matching of labels in the failure mode knowledge base. High-quality reason identification is used to match the core advantages output by the model from the victory mode knowledge base and label the corresponding high-quality reason labels. The structured annotation results include a list of main problem labels and a list of high-quality reason labels. Step S3, Rule Mapping: Based solely on the structured annotation results output in Step S2, and combining the evaluation scale constraint information and the dual knowledge base, a purely numerical fact archive is automatically generated for each model to be evaluated. The fact archive includes: Major Problem Severity Level: Based on the list of major problem tags, match the ratings specified in the failure mode knowledge base, and take the highest rating as the overall major problem severity level of the model; Advantage level of high-quality reasons: Based on the list of high-quality reasons tags, the highest rating specified in the Victory Mode Knowledge Base is taken as the overall advantage level of the model; and this advantage level is only effective when the severity level of the main problem is 0; if the severity level of the main problem is ≥1, the advantage level is forcibly reset to zero. Pass rate level: Based on the severity level of the main problem and the advantage level of the good reasons, the final pass rate level is generated according to the evaluation scale constraint information mapping; the pass rate levels are in absolute order as follows: excellent, qualified, unqualified, and mentally challenged. Step S4, Dual-Model Comparison Evaluation: Based solely on the factual archives of the two models under test output in Step S3, objective numerical differences are calculated to generate dual-model comparison evaluation results; the dual-model comparison evaluation results specifically include: difference in pass rate level, difference in advantages of excellent reasons, difference in severity of major problems, and difference in the number of problems; this step does not perform a final winner determination; Step S5, Multi-dimensional adjudication reasoning: Only the dual-model comparison evaluation results generated in step S4 are called, and truncated adjudication reasoning is performed according to the following macro-to-micro degradation logic. As long as the judgment condition of any priority is met, the judgment is truncated and the judgment result is output: First priority, pass rate ranking determination: Based on the difference in pass rate ranking, the one with the better pass rate ranking wins directly; if both are in the same rank, then proceed to the next priority. Second priority, determination of the superiority level of superior reasons: based on the difference in superiority of superior reasons, the one with the higher superiority level of the effective superior reasons wins; if the difference is zero, then proceed to the next priority. The third priority is the comparison of the severity of the main problems: based on the difference in the severity of the main problems, the one with the lower severity level wins; if the values ​​are the same, then proceed to the next priority. Fourth priority, comparison of the number of issues: Based on the difference in the number of issues, the one with fewer main issue tags wins; Fifth priority, tie-breaker ruling: If the differences in all the above dimensions are indistinguishable, the final judgment is "tie"; The final output includes a referee's explanation of the specific conclusion of the win or loss and the basis for the conclusion.

[0008] Furthermore, the construction of the dual knowledge base and evaluation scale constraint information in step S1 is specifically implemented as follows: Samples with a pass rate of "Excellent" from historical benchmark data are considered excellent samples. Core advantage features are extracted from these samples to construct the victory pattern knowledge base. The core advantage features include pattern name, description, keywords, and the advantage level of the reason for excellence, which is used to quantify the degree of influence of the advantage on the evaluation results. Samples with a pass rate of "Unsatisfactory" and "Incompetent" from historical benchmark data are considered poor samples. Main problem features are extracted from these samples to construct the failure pattern knowledge base. The main problem features include pattern name, description, keywords, and the severity rating of the main problem, which is used to quantify the fatality of the error on the evaluation results.

[0009] Furthermore, the method also includes a dynamic update mechanism for the dual knowledge base and evaluation scale constraint information, specifically: Periodically retrieve newly added precise data and extract candidate winning and losing patterns using a large language model; perform conflict detection and fusion: when identifying duplicate patterns by comparing keyword similarity, retain the pattern with the higher rating, and update conflict patterns based on the patterns in the newly added precise data. Frequency and universality verification: Only candidate patterns that appear more than a preset threshold are formally included in the knowledge base. And regularly scan and remove low-frequency patterns that have not been matched for a long time; The mapping relationship between tags and pass rate levels is recalculated based on the updated knowledge base to achieve iterative updates of the evaluation scale constraint information.

[0010] Furthermore, the evaluation scale constraint information in step S1 includes a scenario-based tolerance threshold, specifically: setting differentiated problem tolerance weights for different evaluation application scenarios; each victory mode and failure mode in the dual knowledge base includes a trigger scenario field; in steps S2 and S3, the system can identify the application scenario to which the current input data belongs, and dynamically load the tolerance threshold of the corresponding scenario to perform feature extraction and archiving mapping.

[0011] Furthermore, step S1 also includes: constructing a fuzzy mapping table from keywords to standard bucket names, normalizing the labels of natural language descriptions to a predefined standard label system, and supporting automatic matching of synonyms and near-synonyms; Design large language model prompts and extract common patterns from the winning case set and the losing case set respectively: for the winning pattern, extract the pattern name, description, keywords and the strength level of the good reasons; for the losing pattern, extract the pattern name, description, keywords and the severity rating of the main problem. The extracted general patterns are stored in JSON format, supporting subsequent automatic matching to achieve automatic extraction of victory and defeat patterns; In the scoring rule configuration file, differentiated label weights and tolerance thresholds are defined by dimension.

[0012] Secondly, this invention provides a multi-model evaluation system based on scale self-calibration and phased adjudication, wherein the system performs the method and includes: Evaluation scale self-calibration module: Based on historical manually calibrated data, it automatically extracts scale features and constructs a dual knowledge base containing a victory mode knowledge base and a failure mode knowledge base, along with evaluation scale constraint information, through a large language model combined with rule reasoning algorithms. Single-model structured annotation module: used to load the evaluation scale constraint information and dual knowledge base, and the large language model performs bidirectional recognition on the answers of each model to be evaluated, outputting structured annotation results containing a list of main question labels and a list of good reasons labels; Rule mapping module: Based on the structured annotation results, combined with the evaluation scale constraint information and dual knowledge base, it automatically maps and generates fact archive files for each model to be evaluated. The fact archive files include the severity level of the main problem, the advantage level of the good reasons, and the pass rate level. Dual-model comparison and evaluation module: Based solely on the numerical fact archives output by the rule mapping module, it performs purely objective numerical difference calculations and generates dual-model comparison and evaluation results that include the difference in pass rate level, the difference in the advantage of excellent reasons, the difference in the severity of major problems, and the difference in the number of problems. This module does not perform a win or loss determination. Multi-dimensional adjudication reasoning module: Used to call the dual-model comparison evaluation results described in the dual-model comparison evaluation module. According to the preset downgrade logic from macro to micro, the order is as follows: pass rate level determination, quality reason advantage level determination, main problem severity comparison, problem quantity comparison, tie fallback determination. It executes truncated adjudication reasoning and outputs the final adjudication result and source explanation.

[0013] Furthermore, the system also includes a knowledge base dynamic update module, which is used to periodically collect newly added precision data and automatically learn the features of the newly added data through a large language model to dynamically update the dual knowledge base and evaluation scale constraint information.

[0014] Thirdly, this invention provides a multimodal large-scale model question-answering evaluation system based on scale self-calibration and phased adjudication. This system fully reuses the overall process, module architecture, and adjudication logic of the evaluation method described above, and customizes a dedicated knowledge base, judgment rules, and boundary constraint criteria for multimodal data features combining images and text. It includes the following: In multimodal scenarios, the precision data input layer supports non-textual modal data (including but not limited to images, videos, and audio) and optional accompanying text commands; the model to be evaluated generates textual or non-textual modal results; this multimodal evaluation system can execute all the aforementioned evaluation methods, and the specific design and rule configuration are as follows: The scale feature and model feature learning layers are based on multimodal precision-calibrated data, extracting and constructing a dual knowledge base and evaluation scale constraint information specific to multimodalities: The system establishes a mapping relationship between the severity level of major issues, the advantage level of good causes, and the pass rate level, and includes built-in boundary judgment criteria for multimodal phenomena. The threshold for setting the pass rate level to "mentally challenged" is triggered when the highest severity level of the extracted major problem is level 5. The judgment experience and special points of level 5 are fatal errors such as "complete fabrication of visual facts" or "serious logical breaks". The threshold for setting the pass rate level to "unqualified" is triggered when the highest extracted major problem severity level is 4. The judgment experience and special points of level 4 are serious problems such as "objectively existing but feature misjudgment" or "incorrect identification of core intent". The threshold for setting the pass rate level to "pass" is triggered when the highest extracted major problem severity level falls between level 1 and level 3. The criteria for judging level 3 and below problems are "the core intent has been met, the main elements have been correctly identified, and there are only flaws in the secondary dimensions". Omission of local non-core background details is judged as level 3, excessive length of related text output is judged as level 2, and poor text layout format is judged as level 1. The strict boundary for classifying the pass rate as "excellent" is: the model is classified as "excellent" if and only if the highest extracted major problem severity level is 0 and at the same time at least one superior feature label with a superior cause advantage level ≥ 1 is extracted; if the model performs as "no major problem" at level 0 but does not trigger any superior label, the critical point is held and it is only classified as "pass". Multimodal Failure Mode Knowledge Base: Contains multiple levels of main problem tags (the following is only an exemplary configuration and is not intended to limit the invention. The specific main problem tags, main problem severity levels, and their mapping relationship to pass rate levels are related to the actual evaluation scenario and can be flexibly adjusted): If the primary issue tag is "poor content quality_visual hallucination", then the severity level of the primary issue is set to level 5. The primary issue is labeled as "No Required Information Provided - Omission of Local Details," meaning that key information in the multimodal data was not fully covered, but no facts were fabricated. The severity level of this primary issue is set to 4. The primary issue is labeled as poor language expression and strong AI-like quality, with a severity level of 2. Multimodal Victory Pattern Knowledge Base: Contains multiple levels of quality reason tags (the following is only an exemplary configuration and is not intended to limit the invention. The specific quality reason tags, quality reason advantage levels, and their mapping relationship to pass rate levels are related to the actual evaluation scenario and can be flexibly adjusted): The high-quality reason is tagged as multimodal_deep cross-reasoning, which accurately combines multimodal data details and text instructions to perform complex logical reasoning. Its high-quality reason advantage level is set to level 4. The "High-Quality Reason" label is "Multimodal_Fine-Grained Feature Capture," which accurately identifies and describes extremely small but crucial details in multimodal data, and its "High-Quality Reason" advantage level is set to 5. The "High-Quality Reason" label is "Multimodal_Accurate Spatial Relationship Analysis," which accurately understands the three-dimensional spatial relative positions of multiple objects in multimodal data, and its "High-Quality Reason" advantage level is set to 5.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1) This invention constructs a progressive thinking chain structure, decomposing the evaluation process into multiple logically progressive stages. The output of each stage serves as the input for the next stage, forming a chain-like reasoning path. Specifically, the progressive thinking chain structure of this application includes four stages: The first stage is single-model structured annotation, where major problem identification and high-quality cause identification are performed independently for each model to be evaluated; the second stage is rule mapping, mapping the annotation results of the first stage to a purely numerical fact archive (including major problem severity level, high-quality cause advantage level, and pass rate level); the third stage is dual-model comparative evaluation, performing objective numerical difference calculations based solely on the mapping results of the second stage; and the fourth stage is multi-dimensional adjudication reasoning, making a final judgment based solely on the difference comparison results of the third stage. There are strict data dependencies between each stage; each subsequent stage can only access the structured output of the previous stage and does not directly access the original model output text, thus achieving complete decoupling of fact extraction and adjudication reasoning at the system architecture level.

[0016] 2) This application implements a highly objective structured fact support mechanism. First, in the single-model structured annotation stage, the main problems and merit reasons output by the model are extracted into a standardized list of tags, each tag coming from a predefined tag system rather than free text descriptions. Second, each tag is assigned a quantified weight attribute (i.e., the advantage level of merit reasons or the severity rating of main problems) through a dual knowledge base (win mode knowledge base and failure mode knowledge base). Third, in the rule mapping stage, the above tags are strictly converted into numerical level indicators (0-5 levels) and pass rate levels for macro-level classification, forming quantifiable and comparable structured data. Finally, in the adjudication stage, the interference of the original text's expression style is completely removed, and the judgment is made solely based on the above purely objective structured archives, greatly improving the objectivity of the evaluation.

[0017] 3) Compared with traditional manual annotation method, fixed rule automatic evaluation method and single-step adjudication model, the core difference of this application is: (1) the evaluation criteria can be automatically learned from the data rather than manually preset; (2) the evaluation process is executed in stages and each stage outputs structured data; (3) the adjudication process pioneered the "macro to micro" multi-priority downgrade adjudication rule, rather than the end-to-end "black box" judgment of a single model.

[0018] 4) Specifically, in terms of engineering applications, this invention brings the following four core breakthroughs: First, dynamic self-calibration of evaluation scales: Based on historical manually calibrated data, it automatically learns evaluation standards, scenario-based tolerance thresholds, and annotation preferences, simultaneously constructing a victory mode knowledge base and a failure mode knowledge base. This transforms implicit human experience into explicit rules, preventing scale drift from the source and ensuring consistency across batches and scenarios. Second, a pioneering five-level progressive adjudication logic from macro to micro: In the final adjudication stage, it innovatively abandons the traditional weighted scoring method and adopts a truncated downgrade reasoning chain of "pass rate level determination → quality reason advantage level determination → main problem severity comparison → main problem quantity comparison → tie as a fallback." The [pass rate level] is used as the highest macro indicator that absorbs all defect and advantage information for the first round of veto, followed by a step-by-step comparison of highlights and flaws, highly consistent with the judgment intuition of human experts, ensuring that the result is indisputable. Third, efficient and structured reuse of human experience: Through the scale self-calibration module and dual knowledge bases, the implicit experience of annotators in identifying core strengths and major problems is transformed into explicit evaluation scale constraint information. This supports regular updates to calibration standards and knowledge bases by adding new precision-calibrated data, enabling continuous accumulation and large-scale reuse of human experience. Fourth, flexible expansion across multiple scenarios: While maintaining the core process architecture, by replacing precision-calibrated data in specific domains and dynamically adjusting the dual knowledge base configuration, the system can be quickly adapted to multimodal, medical, financial, and other vertical domains, as well as cross-language scenarios, broadening its applicability and lifecycle. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the method of the present invention.

[0020] Figure 2 This is a schematic diagram of the evaluation index rule mapping process of the present invention.

[0021] Figure 3 This is a flowchart illustrating the multi-dimensional adjudication reasoning process of the present invention. Detailed Implementation

[0022] To better understand the technical solution of this invention, some core technical terms involved in the embodiments of this invention will be explained and described in advance: 1. "Scale Features" and "Pattern Features": These refer to the implicit experiences of human experts mined from historical manually labeled data during the "knowledge extraction stage" of this invention. Both constitute the original resource library for constructing the evaluation benchmark of this invention: "Scale Features" focus on "cognitive boundaries and tolerance," referring to the subjective tolerance threshold, severity perception, and deduction / addition weight of human annotators for various errors or strengths during evaluation (e.g., in knowledge question-answering scenarios, human experts have zero tolerance for "logical reversals" but a higher tolerance for "a little bit of nonsense"). "Pattern Features" focus on "quality and form at the representational level," referring to the typical semantic templates and patterns in language expression and logical structure of high-quality or low-quality answers (e.g., high-quality answers often have a "general-specific-general structure" pattern; low-quality answers often have a "non-answering" pattern).

[0023] 2. "Evaluation Scale Constraint Information": This refers to the set of executable rules and mapping functions generated after the extracted "scale features" are structurally transformed. If "scale features" are learned human experience, then "evaluation scale constraint information" is the "enforcement standard" formulated and issued based on this. The evaluation scale constraint information specifically includes rules in three dimensions: first, grading standards, which define the hard mapping and limiting relationship between various types of problem labels or advantage labels and the final "macro-level pass rate classification"; second, scenario-based tolerance thresholds, which define different tolerance limits and penalty weights for the same type of defect in different business scenarios (e.g., the difference in severity rating of the same defect in knowledge-based question-and-answer scenarios and command-and-control scenarios); and third, boundary judgment criteria, which record the implicit preferences and critical judgment tendencies of human experts when faced with the coexistence of multi-dimensional features. This information is equivalent to providing a unified and stable "judging yardstick" for large-scale model evaluation.

[0024] 3. "Dual Knowledge Base": This refers to the collective term for the "Victory Mode Knowledge Base" and the "Failure Mode Knowledge Base." The Victory Mode Knowledge Base records high-quality answer feature patterns with significant human preferences; the Failure Mode Knowledge Base records typical major problems containing flaws such as logical fallacies, factual illusions, and safety breaches. The dual knowledge bases, together with the aforementioned evaluation metric constraint information, constitute the benchmark reference constraining the annotation behavior of the large model.

[0025] 4. "Fact Archive": This refers to a set of purely numerical grade indicators that transform the actual performance of the model's responses into a set of grades after stripping away the natural language expression style. The internal data is mapped from structured labels combined with evaluation scale constraint information, and it is the sole input basis for subsequent multi-dimensional objective adjudication.

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of protection of this application.

[0027] like Figure 1 As shown, there is a strict progressive dependency of data structures between the steps of this invention. The specific flow logic is as follows: First, the output generated in step S1 (evaluation scale self-calibration stage) is evaluation scale constraint information and a dual knowledge base. These two outputs serve as global judgment benchmarks and are passed down to all stages as necessary inputs. Next, after loading the above benchmarks in step S2 (single model structured annotation, i.e., fact extraction stage), no final judgment result is output; only purely objective structured annotation results are output.

[0028] Combination Figure 2 The evaluation index rule mapping flowchart of the present invention is as follows: Subsequently, step S3 (rule mapping stage) strictly uses the structured annotation result output in step S2 as the only input parameter, and no longer accesses the original text (i.e. the user's original question and the model's original answer string), and maps the structured annotation result into a numerical fact archive (including the severity level of the main problem, the advantage level of the quality reasons, and the pass rate level). Combination Figure 3 The multi-priority assessment and adjudication flowchart of this invention is as follows: The next stage, step S4 (dual-model comparison assessment stage), performs a comparison solely based on the numerical fact archive output from step S3, outputting the comparison assessment results (i.e., the differences in each level). Finally, step S5 (multi-dimensional adjudication reasoning stage) only calls upon the dual-model comparison assessment results from step S4 for logical deduction, ultimately outputting the adjudication result. Through the aforementioned structured data flow link where "the output of the previous step strictly serves as the sole input of the next step," this invention achieves complete decoupling between fact analysis and adjudication reasoning.

[0029] This invention constructs a fuzzy mapping table (KEYWORD2BUCKET) from keywords to standard bucket names, normalizing natural language descriptions to a predefined standard label system. It supports automatic matching of synonyms and near-synonyms, resolving the issue of inconsistent label representations in labeled data. For example, different annotators may describe the same problem differently; "irrelevant answer" and "incorrect intent recognition" actually refer to the same type of problem. A large language model prompt is designed to extract common patterns from both the winning and losing case sets: for winning patterns, the pattern name, description, keywords, and merit level of the good reasons are extracted; for losing patterns, the pattern name, description, keywords, and severity rating of the main problem are extracted. The extraction results are stored in JSON format, supporting subsequent automatic matching. This achieves automatic extraction of winning and losing patterns, solving the problem that in existing manually labeled data, the descriptions of good reasons and main problems are scattered in the remarks field, making direct structuring difficult. In the scoring rule configuration file, differentiated label weights and tolerance thresholds are defined by dimension. During the rule mapping stage, the corresponding threshold configuration is dynamically loaded according to the dimension labels of the current sample, so as to realize the differentiated setting of the tolerance threshold in different scenarios. This avoids the inconsistency of cross-scenario evaluation due to different tolerances for the same problem in different evaluation dimensions (skills, text generation, knowledge question answering, chat). For example, "redundancy" may be a serious problem in the knowledge question answering scenario, but it may be a minor problem in the text generation scenario.

[0030] The core innovations of this application focus on the following three aspects: Innovation Point 1: Self-calibration Mechanism for Evaluation Scales In existing technologies, evaluation standards are typically manually preset or defined by fixed rules, which cannot adapt to changes in data distribution. This application innovatively proposes an automatic learning mechanism for evaluation scales from historical manually labeled data. Through a large language model combined with rule-based reasoning algorithms, it automatically extracts the mapping relationship between the severity level of major issues and the advantage level of excellent causes to the pass rate level, as well as scenario-based tolerance thresholds and annotation preferences. Simultaneously, it constructs a victory mode knowledge base and a failure mode knowledge base. This mechanism transforms the implicit experience of annotators into explicit rules, achieving dynamic calibration of evaluation standards.

[0031] Innovation Point Two: A Phased Architecture Separating Facts and Rulings. Existing single-step adjudication models determine the winner directly based on the raw output, making them susceptible to interference from presentation style. This application innovatively splits the evaluation into four independent phases: "single-model structured labeling → rule mapping → dual-model comparative evaluation → multi-dimensional adjudication reasoning." The fact extraction phase only outputs structured labels, and the adjudication phase makes judgments based solely on structured data, without accessing the raw output, thus achieving complete decoupling of facts and rulings.

[0032] Innovation Point 3: Dual Knowledge Base-Driven Multi-Priority Adjudication Rules. Existing technologies lack explicit modeling of fatal errors and core strengths. This application innovatively constructs a victory mode knowledge base and a failure mode knowledge base, and proposes a six-level priority adjudication rule (pass rate level determination → quality reason advantage level determination → major problem severity comparison → problem quantity comparison → tie-breaker adjudication), ensuring that the adjudication results are traceable and explainable.

[0033] Example 1: Evaluation of the Fixed Dual Knowledge Base Basic Version This embodiment provides a multi-model evaluation method based on scale self-calibration and phased adjudication, comprising a data input layer, a core processing layer, and a result output layer. This embodiment is the core basic solution, which achieves multi-model comparative evaluation by pre-constructing a fixed dual knowledge base of victory and defeat modes and executing a phased adjudication process.

[0034] In constructing the evaluation data source, conventional random sampling was abandoned in favor of a "Hard-Negative / Positive Mining" mechanism. The evaluation set not only integrates open-source benchmarks (such as the ECom-Chat e-commerce customer service dialogue dataset and the CSDS customer service dataset), but also deeply embeds "extreme long-tail data" from real-world enterprise scenarios—including multi-turn complex intent-jumping dialogues supporting up to 8K contexts, adversarial queries of emotional customer complaints, and boundary test sets that induce the model to generate a business domain knowledge illusion. This creates a high-pressure "stress test" environment, forcing the model under test to expose deep-seated technical defects.

[0035] In the core driving force of the evaluation process, this method employs a schema-constrained verification prompting approach combined with dynamic in-context learning. Specifically, before each evaluation, the system retrieves the most relevant "expert-annotated few-shot" injection prompts that best match the current evaluation scenario through vectorized retrieval. Simultaneously, the evaluation model is forced to decouple "perception" from "decision-making" during the reasoning process—first, in the "feature extraction stage," the system precisely locates text fragments in the output corresponding to the "main problem" or "reason for excellence" as anchor points; then, in the "logical reasoning stage," severity or advantage levels are mapped based on the knowledge base scale. Finally, the underlying layer uses Logit-bias adjustment or a parser to intervene in the decoding process, ensuring that the generated JSON structure field content matches the preset enumeration values ​​in the knowledge base. The specific implementation process is as follows: Step 1: Self-calibration of evaluation scales Based on historical manually calibrated data, a dual knowledge base and evaluation scale constraints are pre-built and fixed as a global judgment benchmark for subsequent evaluations. Specifically, this step includes: Precision data input layer: Collects historical high-quality precision data annotated by domain experts. Each data entry includes user questions, model answers, main question tags, pass rate level (incompetent / unqualified / qualified / excellent), comparison and adjudication results (win / loss / draw) and annotation notes.

[0036] Scale and Pattern Feature Learning Layer: Through a Large Language Model (LLM) combined with rule-based reasoning algorithms, implicit annotation threshold scale features are learned from the historical high-quality precision-calibrated data. Samples with a pass rate of "Excellent" in the precision-calibrated data are designated as high-quality samples, from which core strength features are extracted. These core strength features include the strength name, description, keywords, and strength level (divided into 0 to 5 levels to quantify the strength's impact on the evaluation results). Major problems are extracted from samples with pass rates of "Incompetent" and "Unqualified" in the precision-calibrated data. These major problems include the problem name, description, keywords, and severity rating (divided into 0 levels: No problem, 1 level: Minor, 2 level: Moderate, 3 level: Moderate, 4 level: Severe, 5 level: Fatal, used to indicate the severity of the error's impact on the evaluation results). This forms the pattern features.

[0037] Standard Output Layer: Integrates extracted scale features and pattern features to construct and output a fixed dual knowledge base and evaluation scale constraint information. The evaluation scale constraint information includes: the mapping relationship between the severity level of major problems and the advantage level of excellent causes to the pass rate level (e.g., clarifying the correspondence between the severity level of major problems and the pass rate level), the scenario-based tolerance threshold (distinguishing the problem tolerance of core scenarios and non-core scenarios), and the boundary judgment criteria (clarifying the judgment logic of critical samples).

[0038] The scale features include the mapping relationship between main problem labels / good reason labels and pass rate levels, scenario-based tolerance thresholds, and annotation preferences. The dual knowledge base includes a victory mode knowledge base and a failure mode knowledge base. The evaluation scale constraint information refers to the evaluation criteria automatically extracted and structured from historical precision-calibrated data through a large language model.

[0039] Taking the intelligent customer service application scenario as an example, a comparative evaluation is conducted between the large model A to be tested and a competing large model B in the same field. During the evaluation process, the victory mode knowledge base of this invention can be initialized to include the following entries: Victory Mode Knowledge Base (Keywords in the Victory Mode Knowledge Base example:) "pattern_name" corresponds to the high-quality reason name mentioned earlier, such as "cognitive boundary declaration", "intention". Figure 2"Actively clarifying the meaning," etc. "description" corresponds to the description mentioned above and is used to explain the specific performance of this advantageous model; "keywords" corresponds to the keywords mentioned above and is used to identify the scenarios in which this advantageous model is applicable; Corresponding to the aforementioned advantages of quality reasons, they are divided into 6 levels (Level 0: No quality, Level 1: Weak, Level 2: Relatively weak, Level 3: Moderate, Level 4: Relatively strong, Level 5: Significant), used to quantify the degree of influence of this advantage on the evaluation results.

[0040] Model A and Model B are two models that need to be evaluated. During the evaluation process, the system matches the model outputs with the aforementioned winning patterns one by one. If the model's performance matches a certain winning pattern, the superior cause dominance level of that pattern is recorded. If multiple winning patterns are matched, the model with the highest superior cause dominance level is taken as the overall superior cause dominance level of the model. Example: [ { "pattern_name": "Cognitive Boundary Declaration", "description": "When answering questions of uncertainty, large model A actively declares the knowledge boundaries and provides confidence level hints, while large model B tends to give seemingly certain but actually speculative answers, leading to misjudgments by users." "keywords": ["Timeliness issues", "Niche professional issues", "Predictive issues"], "competitive_contrast": "Large model B is overconfident in these types of issues, while large model A's caution actually earns it a higher credibility score", "type": "win_pattern", "direction": "positive", "weight": "Level 5 Significance" }, { "pattern_name": "meaning" Figure 2 "Actively clarify the meaning" "description": "When there are multiple reasonable interpretations of a user's prompt, Model A can identify ambiguity and provide branching answers or proactively ask questions, while Model B tends to choose the interpretation closest to the literal meaning and answer directly." "keywords": ["ambiguous pronoun reference", "cross-domain homonyms", "implicit premise problem"], "competitive_contrast": "Large model B's single-threaded interpretation leads to irrelevant answers, while large model A's branching approach achieves higher satisfaction", "type": "win_pattern", "direction": "positive", "weight": "Level 4 (Strong)" }, { "pattern_name": "Emotional context adaptation", "description": "Identifies emotional signals in the user's prompt, providing information while matching appropriate tone and empathetic expressions, rather than simply outputting information." Keyword: ["Help-seeking questions", "Description of stressful scenarios", "Negative emotions"] "competitive_contrast": "Large Model B still outputs a cold, impersonal information stream under emotional prompts, while Large Model A shows a significant advantage in emotional adaptation." "type": "win_pattern", "direction": "positive", "weight": "Level 3 Medium" }, { "pattern_name": "Automatic Validation of Counterintuitive Propositions", "description": "When a user's question contains an implicit error, Model A corrects the error before providing the correct guidance, while Model B tends to answer based on the error." Keyword: ["Contains erroneous causal relationships", "Questions based on outdated information", "Anti-scientific premises"], "competitive_contrast": "Model B's compliant response reinforced the user's misperception, while Model A's corrective behavior received a higher professionalism score." "type": "win_pattern", "direction": "positive", "weight": "Level 5 Significance" } ] Failure modes are also evaluated in intelligent customer service application scenarios. The failure mode knowledge base of this invention can be initialized to include the following entries: Keyword explanations in the failure mode knowledge base example: "pattern_name" corresponds to the main problem name mentioned above, such as "overly secure denial", "idempotent problem divergence and degradation", etc. "description" corresponds to the description mentioned above and is used to explain the specific manifestations of this main problem; "keywords" corresponds to the keywords mentioned above and is used to identify scenarios where this main problem is likely to occur. "weight" corresponds to the severity rating of the main problem mentioned above, which is divided into 5 levels (level 0: no major problem, level 1: minor, level 2: mild, level 3: moderate, level 4: severe, level 5: fatal), used to indicate the severity of the impact of the error on the evaluation results.

[0041] During the evaluation process, the system matches the model output against each of the aforementioned failure modes. If the model's performance matches a failure mode, the severity rating of the primary problem for that trigger is recorded. If multiple failure modes are matched, the model with the highest severity rating is taken as the overall severity level of the primary problem for that model. Matching a failure mode with a severity rating of 5 (fatal) will directly result in the model being judged as unsuccessful in the evaluation.

[0042] Example: [ { "pattern_name": "Excessive security denial", "description": "Unnecessary security review rejections were triggered for legitimate and reasonable requests, misjudging harmless creative writing, academic discussions, and hypothetical questions as dangerous requests." "keywords": ["counterfactual assumption problem", "academic issues involving sensitive words but harmless", "discussion of film and television plots"], "competitive_contrast": "An overly sensitive security strategy leads to an artificially high rejection rate. The large model B answers the same questions correctly, causing a precipitous drop in user experience." "type": "loss_trigger", "direction": "negative", "weight": "Level 5 lethal" }, { "pattern_name": "Divergence and Degeneracy of Idempotent Problems", "description": "For questions with a single, definitive answer, multiple contradictory possible answers are output, transforming a 'convergent problem' into a 'divergent problem'." "keywords": ["mathematical calculation problems", "factual problems with standard answers", "logic judgments"], "competitive_contrast": "The model's uncertainty is improperly exposed, causing users to doubt the authenticity of the answers. Larger model B provides more certain outputs on such questions." "type": "loss_trigger", "direction": "negative", "weight": "Level 4 Severe" }, { "pattern_name": "Character Personality Drift", "description": "In multi-turn dialogues, the model exhibits inconsistencies in tone, stance, and knowledge boundaries, with contradictory responses or drastic style shifts." Keyword keywords: ["Long conversation", "Role-playing conversation", "Multi-round follow-up questions across topics"] "competitive_contrast": "Context consistency maintenance failed, users perceived the model as 'unreliable,' and the larger model B performed more stably in long dialogues." "type": "loss_trigger", "direction": "negative", "weight": "Level 4 Severe" } ] Step 2: Structured annotation of a single model For the large model A and large model B to be evaluated, the evaluation scale constraint information and dual knowledge bases are loaded respectively. The large language model then performs main question identification and quality reason identification on the answers of each model. Key Issue Identification: Focusing on core user needs, identify and label key issues from a pre-defined tagging system, prioritizing the matching of fatal flaws in the failure mode knowledge base; High-Quality Reason Identification: Match the core advantages output by the model from the victory mode knowledge base, and label the most significant high-quality reason advantage level.

[0043] Output structured annotation results, including "sample ID, user question, model answer, list of main question tags, list of tags for reasons of excellence, and reasons for annotation".

[0044] Step 3: Rule Mapping Steps Using the structured annotation results output in step S2 as the sole input parameter, and combining the evaluation scale constraint information with the dual knowledge base, a purely numerical fact archive is generated: Major problem severity level mapping: Match the major problem severity rating in the failure mode knowledge base according to the major problem label, and take the highest rating as the overall major problem severity level of the model (level 0 no major problem → level 5 fatal). High-Quality Reason Advantage Level Mapping: The high-quality reason advantage level is determined by matching the high-quality reason tag list with the high-quality reason advantage levels specified in the victory mode knowledge base, and the highest rating is taken as the [High-Quality Reason Advantage Level] of the answer (Level 0: No High-Quality → Level 5: Significant). It is important to note that this level is subject to a zero-defect trigger mechanism: the extracted high-quality reason advantage level is only effective when the answer's [Major Issue Severity Level] is 0; if any major issue exists (severity level ≥ 1), the advantage level is forcibly invalidated / reset to zero and will not participate in subsequent calculations.

[0045] Pass Rate Level Mapping: Based on the aforementioned [Main Problem Severity Level] (levels 0 to 5) and [Quality Reason Advantage Level] (levels 0 to 5), the final [Pass Rate Level] (Idiot / Unqualified / Qualified / Excellent) is generated according to the scale constraint information, and the final pass rate level of the model is automatically generated. The specific mapping rules and triggering conditions are as follows: (1) "Idiot" level judgment (fatal defect veto): If the model's answer hits the preset fatal error, causing its [Main Problem Severity Level] to reach level 5, regardless of whether it has a quality reason, the pass rate level is directly set to "Idiot". This is the highest priority veto. (2) "Unqualified" level judgment: If the model's answer does not have a fatal error, but hits a serious error, causing its [Main Problem Severity Level] to be level 4, the pass rate level is directly set to "Unqualified". (3) "Qualified" level judgment (basic standard): includes two situations: First, the model has minor to moderate problems (the severity level of the major problem is between 1 and 3). In this case, even if the model has a very high quality cause advantage level, its qualification rate level is "qualified". Second, the model has no problems (the severity level of the major problem is 0) and no significant advantages (the quality cause advantage level is 0), and it is also classified as "qualified". (4) "Excellent" level judgment: The premise of excellent evaluation is absolute "zero defects". Only when the model answers no major problems (i.e., the severity level of the major problem is 0) and hits at least one core advantage (i.e., the quality cause advantage level is ≥1) will its qualification rate level be marked as "excellent".

[0046] Step 4: Dual-model comparison and evaluation steps Based on the fact archives of the two models output in step 3 (including the severity level of major issues, the advantage level of good causes, and the pass rate level), a purely objective numerical difference calculation is performed to generate a dual-model comparison evaluation result. This step only outputs the differences of each indicator and does not determine the winner. The dual-model comparison evaluation result includes: Pass rate grade comparison difference: The relative rank difference between the pass rate grades (mentally challenged / unqualified / qualified / excellent) of the two parties; Superior Reason Advantage Difference: The difference in the numerical value of the superior reason advantage level between the two parties; Major Problem Severity Difference: The difference in the severity levels of the major problems between the two parties; Issue Count Difference: The difference in the total number of main issue tags included on both sides.

[0047] Step 5: Multi-dimensional adjudication reasoning steps The dual-model comparison evaluation results generated in step 4 are invoked, and truncated adjudication reasoning is executed according to the following macro-to-micro degradation logic. If any priority condition is met, the decision is truncated and the final adjudication result (win / draw / loss) is output without further judgment: First priority (pass rate level determination): As the highest level of macro comparison, it is directly based on the difference in pass rate level comparison, in the order of "excellent > qualified > unqualified > poor", with the one with a higher pass rate level being the winner; if they are in the same level, it proceeds to the next priority.

[0048] Second priority (adjustment of superior reason advantage level): When both parties have the same pass rate level, the party with the higher superior reason advantage level wins based on the difference in superior reason advantage; if the difference is zero, proceed to the next priority.

[0049] Third priority (comparison of major problem severity): When the advantage level cannot be distinguished (e.g., both are "qualified" or both are "unqualified"), the one with the lower major problem severity level (i.e., the less serious problem) wins based on the difference in major problem severity; if the severity values ​​are also the same, then proceed to the next priority. Fourth priority (comparison of the number of issues): When the severity levels of both parties are the same, the party with fewer major issue tags wins based on the difference in the number of issues; if the numbers are the same, proceed to the next priority.

[0050] Fifth priority (draw as a last resort): If the differences in all the above dimensions are indistinguishable (i.e., the evaluation performance is completely consistent), the final judgment is "draw". The final result output layer will include the judge's explanation of the ruling and the basis for the ruling.

[0051] The advantage of this embodiment is that: (1) Low deployment cost: The dual knowledge bases and scale rules are fixed, and there is no need to continuously invest in updating the precision data; (2) High execution efficiency: eliminates the need for dynamic learning process; (3) Strong consistency of results: After deployment, fixed rules are adopted. The source of the fixed rules is: The evaluation scale constraint information and dual knowledge base of this application are automatically extracted and constructed based on historical manual fine-label data through a large language model combined with rule reasoning algorithm, which can reflect the characteristics and preferences of real annotation behavior.

[0052] (4) This application can achieve scene adaptation because: the evaluation scale constraint information includes a scene-based tolerance threshold, and (skills, text generation, knowledge question answering, chat) sets different problem tolerance levels; each victory mode and failure mode in the dual knowledge base contains a trigger scene field, which can identify the scene to which the current sample belongs and match the corresponding evaluation criteria.

[0053] (5) Existing fixed rules use a linear weighted scoring mechanism, where each indicator is simply added together, which fails to reflect the priority differences of different issues. This application adopts a multi-priority adjudication rule, and the hierarchical adjudication mechanism ensures that serious issues are dealt with first and are not diluted by minor issues.

[0054] (6) While maintaining consistency across batches, this application can achieve differentiated adaptation to different scenarios and ensure that the adjudication results are traceable and interpretable. In contrast, the existing fixed rule automatic evaluation method is rigid and cannot adapt to changes in business scenarios and data distribution shifts. Moreover, the adjudication process lacks interpretability. This application avoids fluctuations caused by dynamic adjustments and achieves high consistency across batches.

[0055] In the above-mentioned intelligent customer service scenario, the evaluation method described in this embodiment has achieved significant quantitative improvement: (1) The evaluation accuracy has been greatly improved: After sampling and verification by human experts, the pass rate level automatically given by this method and the "human alignment rate" of human expert scoring have been greatly improved from 75% in the traditional scoring method to over 92%. (2) The evaluation stability has been enhanced: In the multiple rounds of repeated evaluation experiments on the same batch of test sets, the Kappa consistency coefficient of the evaluation results reached 0.85 (showing high consistency), effectively eliminating the indicator fluctuation caused by the subjective threshold fluctuation of the large model in the traditional evaluation. (3) The efficiency of R&D iteration has been improved: By decomposing the originally general win and lose results into structured main problems and reasons for excellence, the R&D team has shortened the root cause location time of abnormal model performance by an average of about 60%, which has significantly improved the iteration optimization efficiency of the large model in actual business scenarios.

[0056] Example 2: Evaluation of the Dynamically Updated Dual Knowledge Base Optimized Version This embodiment, based on the basic version, adds a dynamic update mechanism for dual knowledge bases and evaluation scales, making it suitable for evaluation scenarios where scenario requirements iterate rapidly and new problems or advantages frequently emerge.

[0057] The dynamic update mechanism refers to the adaptive evolution capability of the dual knowledge base and the evaluation scale constraint information. It can automatically learn new winning and losing modes based on newly added precision samples and incrementally optimize the existing knowledge base, so that the evaluation standard can continuously iterate with changes in business scenarios and data distribution shifts.

[0058] The detailed process of the dynamic update mechanism is as follows, and this process is usually executed automatically at a fixed period (e.g., weekly): The first step is data fusion and preprocessing: The system automatically pulls the newly added precision data within the period and merges and deduplicates it with the historical precision data database to form an updated full dataset.

[0059] The second step is candidate pattern extraction: A large language model is invoked to perform scale feature learning on the newly added precise-labeled data, automatically extracting "candidate winning patterns" and "candidate failing patterns." These candidate patterns encompass the advantages and disadvantages revealed in the new data that may not yet be included in the knowledge base.

[0060] The third step involves incremental updates and optimizations of the two knowledge bases, including: a) Conflict Detection and Fusion: The extracted "candidate patterns" are compared with patterns in the existing knowledge base. If duplicate patterns are found, fusion and update operations are performed, for example, retaining the pattern with higher impact / major issue severity rating, or merging the keyword lists of both. If a candidate pattern conflicts with an existing pattern, the pattern reflected in the newly added precise data is used for coverage updates.

[0061] b) Frequency and Generality Verification: For new patterns that pass conflict detection, the system verifies their frequency of occurrence in the full dataset. Only when a pattern's frequency exceeds a preset threshold (e.g., more than 5 times) is it considered general and formally added to the knowledge base. This effectively filters out accidental noise.

[0062] c) Low-frequency pattern elimination: The system will periodically scan the dual knowledge base and archive or remove "unpopular" patterns that have not been hit by any evaluation samples for a long time (e.g., in the past three months) to keep the knowledge base concise and efficient.

[0063] The fourth step is the iteration of evaluation scale constraint information: Based on the updated knowledge base and the full dataset, the system recalculates scale features such as the label-pass rate level mapping relationship and the scenario-based tolerance threshold.

[0064] After completing the periodic updates, the system loads the latest versions of the dual knowledge base and evaluation scale constraint information for both large models A and B to be evaluated. The subsequent evaluation process strictly follows steps S2 (single model structured annotation), S3 (rule mapping), S4 (dual model comparison and evaluation), and S5 (multi-dimensional adjudication reasoning) from Example 1, completing the entire evaluation chain from structured fact extraction to numerical calculation and then to truncated degradation reasoning, ultimately outputting the adjudication result.

[0065] The advantage of this embodiment is that: (1) It has stronger adaptability, and the dynamic update mechanism enables the knowledge base and scale rules to keep up with the scene iteration; (2) The evaluation accuracy is higher and the rules are more in line with actual evaluation needs; the scalability is better and it can adapt to new scenarios without refactoring the core process. (3) Experience is more fully accumulated, enabling the continuous reuse of artificial knowledge.

[0066] Example 3 This embodiment takes a "multimodal large-scale image-text question-answering evaluation" consisting of multimodal data composed of images and text as input data as an example to illustrate in detail the specific execution process of the present invention in extended scenarios: 1. Knowledge base initialization and scale construction in extended scenarios (corresponding to step S1) In multimodal scenarios, the precision-calibrated data input layer is expanded to support a data format that includes "text" input. Here, "text" input is explicitly defined as meaning that a single input sample must contain at least one image data point, and may include at least one text instruction. The scale feature and model feature learning layers, based on multimodal precision-calibrated data, extract and construct a multimodal-specific dual knowledge base and evaluation scale constraint information. The system establishes a strict mapping relationship between the severity level of major problems, the advantage level of excellent causes, and the pass rate level. It also includes built-in boundary judgment criteria for multimodal phenomena. The following is merely an exemplary configuration and is not intended to limit the invention. The specific major problems, severity levels of major problems, excellent causes, advantage levels of excellent causes, and their mapping relationships to the pass rate level are related to the actual evaluation scenario. Some examples are as follows: The threshold for setting the [Pass Rate Level] to "Idiot": This is triggered when the highest extracted [Major Problem Severity Level] is 5.

[0067] Judgment Experience and Special Points: The core judgment logic for Level 5 is "complete fabrication of visual facts" or "serious logical break." For example, the text describes a subject that does not exist in the image at all (serious visual illusion of creating something out of nothing), or the answer is completely unrelated to the image and instructions (serious irrelevant answer). Such errors are considered fatal.

[0068] The threshold for setting the [Pass Rate Level] to "Unacceptable" is triggered when the highest extracted [Major Problem Severity Level] is 4.

[0069] Determining Experience and Special Cases (the boundary between Level 5 and Level 4): Unlike Level 5's "creating something out of nothing," Level 4 focuses on "objectively existing but misjudged features" or "incorrect recognition of the core intent." For example, the input image does indeed contain a car, but the model identifies a "red sedan" as a "blue truck"; or the model misses the core task required by the text instruction. Although not completely fabricated, it has already caused the answer to lose its core practical value.

[0070] Special traps and experience solidification: In multimodal evaluation, "image-text conflict traps" are often set (i.e., the relevant input text contains misleading information that contradicts the image). If the model blindly follows the text and violates the facts of the image, the "primary problem severity level" of this type of problem is forcibly anchored to level 4 or above, so as to strictly constrain the model's "visual basis anchoring" ability.

[0071] The threshold for setting the [Pass Rate Level] to "Pass" is triggered when the highest extracted [Major Problem Severity Level] falls within the range of 1 to 3.

[0072] Judgment of experience and special points (the boundary between level 4 and level 3): The judgment criteria for level 3 and below are "the core intent has been met, the main visual elements have been correctly identified, and there are only flaws in secondary dimensions." For example, missing local non-core background details (level 3), excessively long related text output (level 2), or poor text formatting (level 1). These problems do not affect the main reading and use.

[0073] The strict boundaries for classifying the pass rate level as "excellent" are as follows: Judgment criteria and special cases: A model can be classified as "Excellent" only if the highest extracted [Major Problem Severity Level] is 0 (i.e., no major problem) and at the same time, at least one [Quality Cause Advantage Level] ≥ 1 is extracted as an advantageous feature label. If the model performs at level 0 ("no major problem") but does not trigger any advantageous labels, the critical point is maintained, and it is only classified as "Acceptable".

[0074] Example of a multimodal failure mode knowledge base: Includes multiple levels of primary problem labels. For example:

Poor Content Quality_Visual Feature Illusion

Primary Problem Severity Level

Failure to Provide Required Information_Omission of Local Details

Primary Problem Severity Level

Poor Language Expression_Strong AI Feeling

Primary Problem Severity Level

[0075] Example of a multimodal victory mode knowledge base: Includes multiple levels of high-quality cause tags. The following is merely an exemplary configuration and is not intended to limit the invention. Specific main issue tags, main issue severity levels, and their mapping to pass rate levels are related to the actual evaluation scenario and can be flexibly adjusted. For example:

Multimodal_Deep Cross-Reasoning

High-Quality Cause Advantage Level

Multimodal_Fine-Grained Feature Capture

High-Quality Cause Advantage Level

Multimodal_Accurate Spatial Relationship Analysis

High-Quality Cause Advantage Level

[0076] 2. Analyzing the evaluation execution chain in extended scenarios (corresponding to steps S2 to S5) Suppose the current input is a user image (image content: 3 cats with 1 red yarn ball next to them), and the user inputs the text: "How many cats are in the picture? What is next to them?".

[0077] Step S2 (Single Model Structured Annotation): The large language model loads a multimodal dual knowledge base and performs fact-checking on the image content and the model's answers.

[0078] The large model A under test answered: "There are 3 cats in the picture, and there is a red ball of yarn next to them. The three cats are arranged in a triangular shape, and the red ball is in the lower right corner of the picture." — The system judged that its answer was completely correct and demonstrated accurate ability to analyze spatial relationships between text and images, and extracted the following tags: [High-quality reasoning list: Multimodal_1. Deep cross-reasoning], [Main problem tag list: No main problem].

[0079] The large model B under test responded: "There are two cats in the picture, they are playing with a blue frisbee, and there is an eagle in the sky." — The system accurately captured multiple factual errors, extracting the [List of High-Quality Reason Tags: None], and extracting a list of [Main Problem Tags: Poor Content Quality_Visual Feature Illusion (fabricated eagle, Level 5); Failure to Provide Required Information_Misjudgment of Core Object (mistaking a ball of yarn for a frisbee, Level 4); Failure to Provide Required Information_Content Error (Incorrect number of cats, Level 4)].

[0080] Step S3 (Rule Mapping): Model A Fact Archive: [Main Problem Severity Level] has a maximum value of 0; [Excellent Reason Advantage Level] is mapped to 4; based on the evaluation scale constraint information, the [Pass Rate Level] is set as "Excellent".

[0081] Model B Fact Archive: Its label list contains one level 5 issue and two level 4 issues. The system extracts the highest value of the severity level of the main issue, and its [Main Issue Severity Level] is mapped to level 5; the [Quality Cause Advantage Level] is level 0; according to the evaluation scale constraint information mapping rules, the [Pass Rate Level] is set as "Incompetent".

[0082] Step S4 (Dual-model comparison and evaluation): Based on the aforementioned archived files, the system performs purely numerical calculations to generate comparison differences across four dimensions: [Comparison of Pass Rate Levels]: Excellent vs. Incompetent (Model A leads by 3 levels); [Main Problem Severity Difference]: 0 minus 5, the difference is -5; [Difference in Advantages Due to Quality Reasons]: 4 minus 0, the difference is +4; [Difference in the number of problems]: Model A contains 0 main problems, and Model B contains 3 main problems (illusion, object misjudgment, content error). Subtracting 3 from 0 results in a difference of -3. This step only outputs the comparison difference, not the winner.

[0083] Step S5 (Multi-dimensional adjudication reasoning): The system makes a decision based on the comparison difference output by S4.

[0084] Entering the first priority (pass rate level determination): The system detected a significant difference in the ranking of the [pass rate level comparison difference] (Model A is "excellent", Model B is "incompetent"). Based on the absolute ranking order of "excellent > qualified > unqualified > incompetent", the reasoning for subsequent priorities is directly truncated (no need to compare severity or number of problems), and Model A is determined to win.

[0085] The system outputs a ruling explanation: "Model A is determined to win. Reason: Model B triggered a total of 3 major issues, including [Poor Content Quality_Visual Feature Illusion] (Level 5), with the highest [Major Issue Severity Level] reaching Level 5, triggering the severe illusion threshold, and its pass rate level is 'Idiot'; while Model A has no issues and hits the deep cross-reasoning advantage, with a pass rate level of 'Excellent'. Due to the absolute difference in the pass rate level, Model A wins." 3. Support for other derivative and extended scenarios Based on the same system architecture and decoupling logic, this invention can also be horizontally extended to the following application scenarios: Domain-specific model evaluation (e.g., healthcare, finance): No changes are needed to the core adjudication code in steps S3-S5; only the import of specialized, high-precision data for that domain in step S1 is required for scale self-calibration. For example, in the failure mode knowledge base for healthcare scenarios, the [Main Problem Severity Level] for specific tags such as "Violation of treatment guidelines" and "Giving a definitive prescription" can be configured as a Level 5 fatal error.

[0086] Agent (Intelligent Agent) Behavior Path Evaluation: The traditional "model response text" is replaced with "Agent's API call sequence or action flow". The structured annotation step S2 is transformed into feature extraction of action rationality and call redundancy. The winning mode can be configured with advantage labels such as "optimal tool call path", and the failure mode can be configured with severity labels such as "entered API infinite loop" or "illegal parameter passing".

[0087] Cross-language model evaluation: A language adaptation layer is added before the precision data input layer. The original inputs from different languages ​​are transformed into a unified representation through semantic alignment. The dual knowledge base and adjudication logic under the general language framework are reused to achieve objective comparison across languages.

[0088] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0089] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A multi-model evaluation method based on scale self-calibration and phased adjudication, characterized in that, The method includes the following strictly progressive and data-decoupled steps: Step S1, Evaluation Scale Self-calibration: Based on historical manually calibrated data, the scale features and pattern features are automatically extracted through a large language model combined with a rule-based reasoning algorithm to construct a dual knowledge base and evaluation scale constraint information; the dual knowledge base includes a victory mode knowledge base and a failure mode knowledge base; the evaluation scale constraint information includes scenario-based tolerance thresholds, boundary judgment criteria, and the mapping relationship between the severity level of major problems and the advantage level of excellent reasons to the pass rate level; Step S2, Single-model structured annotation: For multiple large models to be evaluated, the evaluation scale constraint information and dual knowledge bases are loaded respectively. The large language model performs fact extraction on the original answers of each model and outputs only the structured annotation results. The fact extraction includes two aspects: main problem identification and high-quality reason identification. Main problem identification is used to identify and label the main problems from the preset label system, prioritizing the matching of labels in the failure mode knowledge base. High-quality reason identification is used to match the core advantages output by the model from the victory mode knowledge base and label the corresponding high-quality reason labels. The structured annotation results include a list of main problem labels and a list of high-quality reason labels. Step S3, Rule Mapping: Based solely on the structured annotation results output in Step S2, and combining the evaluation scale constraint information and the dual knowledge base, a purely numerical fact archive is automatically generated for each model to be evaluated. The fact archive includes: Major Problem Severity Level: Based on the list of major problem tags, match the ratings specified in the failure mode knowledge base, and take the highest rating as the overall major problem severity level of the model; Advantage level of high-quality reasons: Based on the list of high-quality reasons tags, the highest rating specified in the Victory Mode Knowledge Base is taken as the overall advantage level of the model; and this advantage level is only effective when the severity level of the main problem is 0; if the severity level of the main problem is ≥1, the advantage level is forcibly reset to zero. Pass rate level: Based on the severity level of the main problem and the advantage level of the good reasons, the final pass rate level is generated according to the evaluation scale constraint information mapping; the pass rate levels are in absolute order as follows: excellent, qualified, unqualified, and mentally challenged. Step S4, Dual-model comparison and evaluation: Based solely on the fact archives of the two models under test output in step S3, perform objective numerical difference calculations to generate dual-model comparison and evaluation results; The results of the dual-model comparison and evaluation specifically include: the difference in pass rate level, the difference in advantages of excellent reasons, the difference in the severity of major problems, and the difference in the number of problems; this step does not perform a final victory or defeat determination; Step S5, Multi-dimensional adjudication reasoning: Only the dual-model comparison evaluation results generated in step S4 are called, and truncated adjudication reasoning is performed according to the following macro-to-micro degradation logic. As long as the judgment condition of any priority is met, the judgment is truncated and the judgment result is output: First priority, pass rate ranking determination: Based on the difference in pass rate ranking, the one with the better pass rate ranking wins directly; if both are in the same rank, then proceed to the next priority. Second priority, determination of the superiority level of superior reasons: based on the difference in superiority of superior reasons, the one with the higher superiority level of the effective superior reasons wins; if the difference is zero, then proceed to the next priority. The third priority is the comparison of the severity of the main problems: based on the difference in the severity of the main problems, the one with the lower severity level wins; if the values ​​are the same, then proceed to the next priority. Fourth priority, comparison of the number of issues: Based on the difference in the number of issues, the one with fewer main issue tags wins; Fifth priority, tie-breaker ruling: If the differences in all the above dimensions are indistinguishable, the final judgment is "tie"; The final output includes a referee's explanation of the specific conclusion of the win or loss and the basis for the conclusion.

2. The method according to claim 1, characterized in that, The construction of the dual knowledge base and evaluation scale constraint information in step S1 is specifically implemented as follows: Samples with a pass rate of "Excellent" in the historical benchmark data are used as excellent samples. Core advantage features are extracted from these samples to construct the victory pattern knowledge base. The core advantage features include pattern name, description, keywords, and the advantage level of the reason for excellence. The advantage level of the reason for excellence is used to quantify the degree of influence of the advantage on the evaluation results. Samples with a pass rate of "Unsatisfactory" and "Incompetent" in the historical benchmark data are used as poor samples. Main problem features are extracted from these samples to construct the failure pattern knowledge base. The main problem features include pattern name, description, keywords, and the severity rating of the main problem. The severity rating of the main problem is used to quantify the fatality of the error on the evaluation results.

3. The method according to claim 1, characterized in that, The method also includes a dynamic update mechanism for the dual knowledge base and evaluation scale constraint information, specifically: Periodically retrieve newly added precise data and extract candidate winning and losing patterns using a large language model; perform conflict detection and fusion: when identifying duplicate patterns by comparing keyword similarity, retain the pattern with the higher rating, and update conflict patterns based on the patterns in the newly added precise data. Frequency and universality verification: Only candidate patterns that appear more than a preset threshold are formally included in the knowledge base. And regularly scan and remove low-frequency patterns that have not been matched for a long time; The mapping relationship between tags and pass rate levels is recalculated based on the updated knowledge base to achieve iterative updates of the evaluation scale constraint information.

4. The method according to claim 1, characterized in that, The evaluation scale constraint information in step S1 includes a scenario-based tolerance threshold, specifically: setting differentiated problem tolerance weights for different evaluation application scenarios; each victory mode and failure mode in the dual knowledge base includes a trigger scenario field; in steps S2 and S3, the system can identify the application scenario to which the current input data belongs, and dynamically load the tolerance threshold of the corresponding scenario to perform feature extraction and archiving mapping.

5. The method according to claim 1, characterized in that, Step S1 also includes: constructing a fuzzy mapping table from keywords to standard bucket names, normalizing the labels of natural language descriptions to a predefined standard label system, and supporting automatic matching of synonyms and near-synonyms; Design large language model prompts and extract common patterns from the winning case set and the losing case set respectively: for the winning pattern, extract the pattern name, description, keywords and the strength level of the good reasons; for the losing pattern, extract the pattern name, description, keywords and the severity rating of the main problem. The extracted general patterns are stored in JSON format, supporting subsequent automatic matching to achieve automatic extraction of victory and defeat patterns; In the scoring rule configuration file, differentiated label weights and tolerance thresholds are defined by dimension.

6. A multi-model evaluation system based on scale self-calibration and phased adjudication, characterized in that, The system for performing the method according to any one of claims 1-5 comprises: Evaluation scale self-calibration module: Based on historical manually calibrated data, it automatically extracts scale features and constructs a dual knowledge base containing a victory mode knowledge base and a failure mode knowledge base, along with evaluation scale constraint information, through a large language model combined with rule reasoning algorithms. Single-model structured annotation module: used to load the evaluation scale constraint information and dual knowledge base, and the large language model performs bidirectional recognition on the answers of each model to be evaluated, outputting structured annotation results containing a list of main question labels and a list of good reasons labels; Rule mapping module: Based on the structured annotation results, combined with the evaluation scale constraint information and dual knowledge base, it automatically maps and generates fact archive files for each model to be evaluated. The fact archive files include the severity level of the main problem, the advantage level of the good reasons, and the pass rate level. Dual-model comparison and evaluation module: Based solely on the numerical fact archives output by the rule mapping module, it performs purely objective numerical difference calculations and generates dual-model comparison and evaluation results that include the difference in pass rate level, the difference in the advantage of excellent reasons, the difference in the severity of major problems, and the difference in the number of problems. This module does not perform a win or loss determination. Multi-dimensional adjudication reasoning module: Used to call the dual-model comparison evaluation results described in the dual-model comparison evaluation module. According to the preset downgrade logic from macro to micro, the order is as follows: pass rate level determination, quality reason advantage level determination, main problem severity comparison, problem quantity comparison, tie fallback determination. It executes truncated adjudication reasoning and outputs the final adjudication result and source explanation.

7. The system according to claim 5, characterized in that, The system also includes a knowledge base dynamic update module, which is used to periodically collect newly added precision data and automatically learn the features of the newly added data through a large language model to dynamically update the dual knowledge base and evaluation scale constraint information.

8. A multimodal large model question-answering evaluation system based on scale self-calibration and phased adjudication, characterized in that, Performing the method according to any one of claims 1-5 includes the following: In multimodal scenarios, the precision data input layer supports non-text modal data and optional accompanying text instructions; the non-text modal data includes at least one of images, videos, and audio, and the model to be evaluated generates text or non-text modal results; The scale feature and model feature learning layers are based on multimodal precision-calibrated data, extracting and constructing a dual knowledge base and evaluation scale constraint information specific to multimodalities: The system establishes a mapping relationship between the severity level of major issues, the advantage level of good causes, and the pass rate level, and includes built-in boundary judgment criteria for multimodal phenomena. The threshold for setting the pass rate level to "idiotic" is triggered when the highest severity level of the extracted major problem is level 5. The judgment experience and special points of level 5 include fatal errors such as "complete fabrication of visual facts" or "serious logical breaks". The threshold for setting the pass rate level to "unqualified" is triggered when the highest extracted major problem severity level is 4. The judgment experience and special points of level 4 include serious problems such as "objectively existing but feature misjudgment" or "incorrect identification of core intent". In multimodal evaluation, a "text-image conflict trap" is set, in which the relevant input text contains misleading information that contradicts the image. If the model blindly follows the text and violates the facts of the image, the severity level of the main problem of this type of problem is forcibly anchored to level 4 or above, so as to strictly constrain the model's "visual basis anchoring" ability. The threshold for setting the pass rate level to "pass" is triggered when the highest extracted major problem severity level falls between level 1 and level 3. The criteria for judging level 3 and below problems are "the core intent has been met, the main visual elements are correctly identified, and there are only flaws in the secondary dimensions". Omission of local non-core background details is judged as level 3, excessive length of related text output is judged as level 2, and poor text layout format is judged as level 1. The strict boundary for classifying the pass rate as "excellent" is: it is classified as "excellent" if and only if the highest extracted major problem severity level is 0 and at the same time at least one superior feature label with a superior cause advantage level ≥ 1 is extracted; if the model performs as 0 "no major problems" but does not trigger any superior label, the critical point is held and it is only classified as "pass". Multimodal failure mode knowledge base: contains multiple levels of major problem tags, with the severity level of major problems set from 0 to 5, where level 0 is no major problem, level 1 is minor, level 2 is relatively minor, level 3 is moderate, level 4 is severe, and level 5 is fatal. Multimodal Victory Pattern Knowledge Base: Contains multiple levels of quality reason tags, with quality reason advantage levels set from 0 to 5, where level 0 is no quality, level 1 is weak, level 2 is relatively weak, level 3 is moderate, level 4 is relatively strong, and level 5 is significant.

Images