Evaluation method based on field large model illusion

Abstract

The invention relates to a field large model illusion-based assessment method, which comprises the following steps of S1, dividing the capabilities of a field large model to be assessed into three major classes, namely professional knowledge capability, general capability and performance, and carrying out detailed definition on each class of capabilities to construct a field large model assessment system; s2, collecting multi-format document data of a target domain, generating domain knowledge question-answer pairs based on the document slices by using a large language model with reasoning ability, and forming an initial evaluation data set; s3, based on the initial evaluation data set, adopting a model-manual double-layer screening mechanism to obtain a high-quality evaluation data set; and S4, comparing and evaluating the output answers of the large model of the field to be evaluated with the standard answers in the high-quality evaluation data set. According to the method, the problems of single evaluation dimension, insufficient automatic evaluation reliability, difficulty in constructing a high-quality evaluation data set and the like in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of large model evaluation technology, and specifically to an evaluation method based on the illusion of a domain large model. Background Technology

[0002] As Large Language Models (LLMs) are increasingly applied in military, medical, financial, and other fields and industries, they are reshaping the landscape of various applications and demonstrating enormous transformative potential. Measuring the optimization level of large models during training and ensuring their reliability during use are core objectives of large model evaluation research. Model evaluation not only helps users understand the model's performance and limitations but also guides developers in optimizing the model and ensuring its safety. Effective evaluation methods and mechanisms can make large models more accurate, secure, efficient, and more adaptable to diverse application scenarios. Model "hallucination" refers to inaccurate facts generated by the model that do not match the input information or real-world knowledge. This is one of the main obstacles to the high reliability of large models, directly affecting the model's credibility and the authenticity of its output. Therefore, a systematic, scientific, and quantifiable evaluation of the model's hallucination capability is crucial.

[0003] Current research on domain-specific large-scale model illusion assessment is still in its early exploratory stage. Mainstream assessment methods heavily rely on models based on Natural Language Inference (NLI) or traditional Natural Language Processing (NLP) metrics, such as ROUGE and BLEU. These methods have significant limitations: First, the reliability of automated assessment methods themselves needs to be verified, and they struggle to accurately capture complex logical fallacies and factual errors in open-domain texts. Second, the assessment dimensions are singular, focusing primarily on factual correctness and lacking a comprehensive quantitative examination of multiple dimensions such as the logical coherence and information completeness of generated content. Third, the assessment process lacks systematicity and is usually not integrated with a structured domain capability assessment system, resulting in fragmented assessment results that fail to fully reflect the model's true performance in a specific domain.

[0004] Furthermore, constructing high-quality, domain-specific evaluation datasets is a major challenge. Existing methods often rely on manual compilation or simple screening from general corpora, resulting in low efficiency, high cost, narrow coverage, and difficulty in ensuring the quality and diversity of question-answer pairs. There is a lack of an automated, standardized process for constructing high-quality domain-specific question-answer pairs.

[0005] For example, patent CN120218179A discloses a large-scale model fine-tuning method for detecting hallucinations based on high belief levels. It identifies difficult-to-correct hallucinations by calculating the model's "belief level" in incorrect answers. However, its evaluation relies on the uncertainty estimation of the evaluated model itself, representing a "first-party" evaluation perspective without introducing an independent, multi-dimensional evaluation model. Patent CN118886421A proposes mitigating hallucinations through result verification, but its verification primarily focuses on factual consistency and lacks a complete systematic solution encompassing system construction, dataset generation, and multi-dimensional quantitative evaluation.

[0006] Therefore, there is a lack of a complete solution in the existing technology that can systematically, multidimensionally, and quantitatively evaluate the illusion of large domain models and efficiently construct high-quality domain evaluation datasets. Summary of the Invention

[0007] In view of the above-mentioned technical problems, this invention proposes an evaluation method based on the domain large model illusion. By constructing a structured evaluation system, standardizing the construction of a high-quality evaluation dataset, and introducing an independent judge model evaluation method based on multi-dimensional quantitative scoring, a comprehensive, accurate, and repeatable evaluation of the domain large model illusion can be achieved, solving the problems of single evaluation dimension, insufficient reliability of automated evaluation, and difficulty in constructing high-quality evaluation datasets in existing technologies.

[0008] The technical solution to the technical problem of this invention is: an evaluation method based on the illusion of a large domain model, comprising the following steps:

[0009] Step S1: Divide the capabilities of the domain model to be evaluated into three categories: professional knowledge capabilities, general capabilities, and performance capabilities, and further define each type of capability in detail to construct a domain model evaluation system.

[0010] Step S2: Collect multi-format document data in the target domain, and use a large language model with reasoning capabilities to generate domain knowledge question-answer pairs based on the document slices to form an initial evaluation dataset;

[0011] Step S3: Based on the initial evaluation dataset, a model-human two-layer screening mechanism is used to obtain a high-quality evaluation dataset;

[0012] Step S4: Compare and evaluate the output answer of the large model in the domain to be evaluated with the standard answer in the high-quality evaluation dataset.

[0013] According to one technical solution of the present invention, in step S1, the professional knowledge capabilities include military domain capabilities, aerospace domain capabilities, and command invocation capabilities;

[0014] The general abilities mentioned include world knowledge, reasoning ability, and language ability;

[0015] The performance characteristics include alignment capability, robustness, and efficiency.

[0016] According to one technical solution of the present invention, step S2 specifically includes:

[0017] Step S21: Collect document data in the target domain and convert it into Markdown format.

[0018] Step S22: Based on the chapter or title structure of the Markdown format document, cut the document into multiple semantically complete text slices;

[0019] Step S23: Input the text slice into a preset large language model to generate knowledge question-answer pairs related to the slice content. The question-answer pairs are stored in JSON format.

[0020] Step S24: Calculate the text vectors of the generated question-answer pairs, and deduplicate question-answer pairs with similarity exceeding a preset threshold based on vector similarity to form an initial evaluation dataset.

[0021] According to one technical solution of the present invention, step S3 specifically includes:

[0022] A trained text classification model is used to automatically filter the question-answer pairs in the initial evaluation dataset, removing low-quality question-answer pairs.

[0023] Based on the preset question-answer pair quality evaluation criteria, the question-answer pairs selected by the model are manually screened to obtain a high-quality evaluation dataset.

[0024] The text classification model is the Fasttext model, which is trained on question-and-answer pairs labeled with high-quality and low-quality tags.

[0025] The preset quality evaluation criteria for the question-and-answer pairs include at least one of the following dimensions: language fluency, question-and-answer relevance, question-and-answer effectiveness, question-and-answer completeness, question-and-answer accuracy, and format standardization.

[0026] According to one technical solution of the present invention, step S4 specifically includes:

[0027] The judge model is constructed to evaluate prompt words. The prompt words require the judge model to first extract and structure key information from the output answer and the standard answer respectively, and then score the output answer from three dimensions: factual consistency, logical consistency and information completeness. Based on the scoring results of the three dimensions, a comprehensive model illusion judgment score is calculated.

[0028] According to one technical solution of the present invention, in step S4, the factual consistency score is... Calculated using the following formula:

[0029] ,

[0030] in, Score for factual consistency. n represents the number of key information items in the answer. For the factual difference degree of the i-th key information, when the key information is a number or a parameter, When the key information is text, , For key information in the reference answer, This provides key information for the output results.

[0031] According to a technical solution of the present invention, in step S4, the information integrity score is... Calculated using the following formula:

[0032] ,

[0033] in, Scoring for information completeness, , To determine the number of key information points in the reference answer, This represents the number of key information items in the output.

[0034] According to one technical solution of the present invention, in step S4, the logical self-consistency score LC is directly scored by the referee model within the range of 0 to 1 based on its understanding of the internal logical coherence of the output answer.

[0035] According to one technical solution of the present invention, in step S4, the model illusion judgment score MHJS is obtained by weighted sum or arithmetic mean of fact consistency score FC, logical self-consistency score LC and information integrity score IC.

[0036] According to one aspect of the present invention, an evaluation system based on the illusion of a large domain model is proposed for implementing the method as described in any of the above technical solutions, comprising:

[0037] The evaluation system construction module is used to divide the capabilities of the domain model to be evaluated into three categories: professional knowledge capabilities, general capabilities, and performance capabilities, and to further define each type of capability in detail to construct an evaluation system for the domain model.

[0038] The dataset construction module is used to collect multi-format document data in the target domain, convert them into a structured text format and slice them; use a large language model to generate domain knowledge question-answer pairs based on the document slices, and perform deduplication on the generated question-answer pairs to form an initial evaluation dataset;

[0039] The data filtering module is used to automatically filter the initial evaluation dataset using a trained text classification model, and to manually filter the dataset based on preset question-and-answer criteria to obtain a high-quality evaluation dataset.

[0040] The illusion assessment module is used to compare the output answer of the large model in the domain to be assessed with the standard answer in the high-quality assessment dataset based on the judge model. The assessment includes: requiring the judge model to extract key information and summarize the structured information of the answer through assessment prompts, and scoring it from three dimensions: factual consistency, logical consistency and information completeness, so as to calculate a comprehensive model illusion judgment score.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention provides an evaluation method based on domain-wide model illusion. By pre-constructing an evaluation system that covers professional knowledge, general capabilities, and performance, it provides a structured framework for illusion evaluation, making the evaluation no longer a fragmented test, but a systematic measurement of the model's performance in different dimensions, resulting in more comprehensive and instructive evaluation conclusions.

[0043] This invention automates and standardizes the construction of evaluation datasets by standardizing and structurally slicing multi-format documents, generating question-answer pairs in batches using a large model, and combining this with vector deduplication technology. This method efficiently generates a large number of question-answer pairs covering a wide range of domain knowledge points, and combines a two-layer screening mechanism (model initial screening + manual fine screening) to ensure the high quality, high relevance, and low redundancy of the final dataset, solving the problems of high cost and low efficiency in manual dataset construction.

[0044] This invention introduces an independent "judge model" for third-party evaluation. By designing evaluation prompts, it mandates that the judge model summarize the answers in points and extract keywords before scoring, making the evaluation process more focused and structured. Furthermore, this invention proposes a quantitative scoring system based on three complementary dimensions: factual consistency (quantitative matching degree), logical consistency (evaluation coherence), and information completeness (quantitative coverage). It also provides specific calculation formulas (such as FC and IC), enabling hallucination evaluation to move from qualitative to quantitative methods. The results are more objective, refined, and repeatable, significantly improving the accuracy and credibility of the evaluation.

[0045] This invention establishes a complete automated evaluation process, encompassing evaluation system design, dataset construction, automated screening, and multi-dimensional quantitative scoring. Combined with the generation of a final comprehensive evaluation report, it achieves a closed loop in the evaluation process, significantly improving the efficiency and standardization of domain-specific large-scale model illusion evaluation. Attached Figure Description

[0046] Figure 1 The schematic diagram illustrates a flowchart of an evaluation method based on the illusion of a large domain model in one embodiment of the present invention;

[0047] Figure 2 This illustration shows a flowchart of the evaluation dataset construction in one embodiment of the present invention;

[0048] Figure 3 The flowchart and actual case illustrate the hallucination assessment method of the referee model in one embodiment of the present invention. Detailed Implementation

[0049] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0050] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0051] like Figures 1 to 3 As shown, this invention provides an evaluation method based on the illusion of a large domain model. When evaluating the capabilities of large models, a domain-wide large model evaluation system is constructed. This system comprehensively analyzes model capabilities and evaluation dimensions, dividing the capabilities of the large domain model into three categories: professional knowledge capabilities, general capabilities, and performance, thus providing a comprehensive evaluation of model capabilities. When evaluating the model's professional knowledge capabilities, the content generated by the model is comprehensively scored across multiple dimensions. The evaluation method is highly sensitive to key information generated by the model. Simultaneously, during the construction of the evaluation dataset, the efficiency of constructing the evaluation dataset is improved by optimizing the data processing and filtering process. The main steps include:

[0052] Step S1: Confirm the domain-wide model evaluation system

[0053] The capabilities of the domain-wide model to be evaluated are divided into three categories: professional knowledge capabilities, general capabilities, and performance capabilities. Each category is further defined in detail to construct an evaluation system for the domain-wide model.

[0054] Professional knowledge capability refers to the level of knowledge mastery and application of a model within a specific vertical domain. This can be further refined into: military domain (such as military reconnaissance, operational theory, and equipment knowledge), aerospace domain (such as satellite applications and aerospace equipment), and command invocation domain (such as the ability to understand and execute complex mission commands).

[0055] General capabilities refer to the model's fundamental capabilities that are independent of its domain. These can be further refined into: world knowledge (understanding of general facts), reasoning ability (logical reasoning, mathematical reasoning, etc.), and language ability (text comprehension, generation, translation, etc.).

[0056] Performance typically refers to the non-functional metrics of a model during application. It can be further refined into: alignment capability (whether it complies with ethical and safety standards), robustness (stability against adversarial inputs or noise), and efficiency (response speed and resource consumption).

[0057] By utilizing a domain-wide model assessment system, assessment objectives can be clearly defined and broken down, allowing subsequent illusion assessments to be linked to specific capability dimensions, thereby generating more insightful assessment reports.

[0058] Step S2: Evaluation Dataset Construction

[0059] Combination Figure 2 Collect multi-format document data in the target domain, and use a large language model with reasoning capabilities to generate domain knowledge question-answer pairs based on the document slices to form an initial evaluation dataset, specifically including:

[0060] Step S21: Collect document data in the target domain and convert it into Markdown format.

[0061] Collect raw data from professional books, technical documents, and authoritative databases in the target field. This data may be in various formats such as PDF, Word, and TXT. Use format conversion tools or scripts to convert all documents into a clear, easily processed Markdown format.

[0062] Due to inconsistent data formats and different acquisition channels, document classification is required before format conversion.

[0063] Step S22: Based on the chapter or title structure of the Markdown format document, cut the document into multiple semantically complete text slices;

[0064] Traverse the Markdown document and, based on its inherent hierarchical markers such as chapters and headings, cut the long document into multiple semantically independent and complete text slices. This ensures that, during the generation of question-and-answer questions using a large model, the input token does not exceed the model's input limits and that the question-and-answer content remains specific and complete.

[0065] Step S23: Input the text slice into a preset large language model to generate knowledge question-answer pairs related to the slice content. The question-answer pairs are stored in JSON format.

[0066] The text slices obtained in the previous step are used as context and input into a pre-selected large language model with strong reasoning capabilities (such as DeepSeek-R1-0524, Qwen 3-235B-A22B, etc.). Simultaneously, carefully designed prompts are provided to the model, instructing it to generate a relevant question-and-answer pair based on the given text slices (e.g., "Question: ...? Answer: ..."). All question-and-answer pairs generated by the model are stored in structured JSON format (containing fields such as "context", "question", and "answer").

[0067] Alternatively, the question-and-answer format could be transformed into a multiple-choice format. For example, the knowledge-based assessment dataset could be divided into two types: multiple-choice and open-ended questions. Multiple-choice questions would measure the model's mastery of professional knowledge, while open-ended questions would measure the model's generation performance. The ratio of multiple-choice to open-ended questions would be 1:1. Typically, multiple-choice questions would be used as objective questions, and open-ended questions as subjective questions. The objective assessment dataset would be used to score the model's various capabilities according to different categories. The subjective assessment dataset would be used, employing a referee-based model illusion assessment method to score the model's various capabilities according to different categories.

[0068] Step S24: Calculate the text vectors of the generated question-answer pairs, and deduplicate question-answer pairs with similarity exceeding a preset threshold based on vector similarity to form an initial evaluation dataset.

[0069] Since the generated question-answer pairs may contain a large amount of semantically similar or repetitive content, deduplication is necessary. Vector similarity is used to remove identical or similar question-answer pairs and multiple-choice questions. The vector similarity threshold is set to 0.9 to reduce the size of the dataset, improve evaluation efficiency, and export the question-answer pair dataset as a JSON file to form the initial evaluation dataset.

[0070] Step S3: Evaluate the quality of the dataset and screen it.

[0071] Based on the initial evaluation dataset, a high-quality evaluation dataset was obtained by using a model-human two-layer screening mechanism.

[0072] First, an automatic initial screening of the model is performed. A pre-trained Fasttext text classification model is used. This model is trained on samples labeled with "high-quality" and "low-quality" questions and answers by humans. It can quickly perform binary classification on a massive number of initial question-answer pairs, automatically filtering out most obviously low-quality samples, which greatly reduces the workload of subsequent manual work.

[0073] Then, a manual screening process is conducted. Domain experts or trained annotation specialists review the results after model screening based on question-answer pair quality evaluation criteria. These criteria define the conditions that high-quality question-answer pairs must meet, considering multiple dimensions such as language fluency, question-answer relevance, question-answer validity, question-answer completeness, question-answer accuracy, and format conformity. Only question-answer pairs that pass manual review are included in the final high-quality evaluation dataset.

[0074] The evaluation dataset is set at 10% of the total training data volume to ensure comprehensive evaluation.

[0075] Step S4: Conduct multi-dimensional hallucination assessment based on the referee model.

[0076] The evaluation involves comparing the output answer of the large model in the domain to be evaluated with the standard answers in the high-quality evaluation dataset. Specifically, this includes:

[0077] The judge model is constructed to evaluate prompt words. The prompt words require the judge model to first extract and structure key information from the output answer and the standard answer respectively, and then score the output answer from three dimensions: factual consistency, logical consistency and information completeness. Based on the scoring results of the three dimensions, a comprehensive model illusion judgment score is calculated.

[0078] First, the questions are extracted from the high-quality evaluation dataset and input into the domain-specific large model to be evaluated, thereby obtaining its output answer.

[0079] Then, a dedicated prompt word for the judging model is constructed. This prompt word contains key instructions: (i) Extraction and summarization: The judging model is required to summarize the output answer of the model to be evaluated and the standard answer in the dataset according to the same dimensions, and extract key information or parameters, thereby structuring the free text answer for easier subsequent accurate comparison. (ii) Multi-dimensional scoring: The judging model is required to independently score the output answer from the following three dimensions based on the extracted information:

[0080] (1) Factual consistency: Evaluate the degree of matching between the key facts, data, and parameters in the output answer and the standard answer.

[0081] (2) Logical self-consistency: Evaluate whether the logic within the output answer is coherent and reasonable, and whether there are any self-contradictions.

[0082] (3) Information completeness: Evaluate whether the output answer covers the core information points in the standard answer.

[0083] Next, the judge model (or another high-performance general or domain-specific model) performs an evaluation based on the prompt word and outputs scores in three dimensions.

[0084] Finally, a comprehensive Model Hallucination Judgment Score is calculated according to a predefined formula. For example, it can be achieved using... FC is the factual consistency score, LC is the logical consistency score, and IC is the information integrity score.

[0085] This invention overcomes the problems of existing evaluation methods being fragmented, one-sided, and highly subjective, and achieves systematic, automated, and objective quantification of evaluation work, significantly improving the efficiency and credibility of domain-wide model illusion evaluation.

[0086] In some embodiments of the present invention, the evaluation method based on the domain large model illusion further includes:

[0087] Step S5: Generate a comprehensive evaluation report.

[0088] In this embodiment, all the above evaluation results are summarized, and the report content typically includes:

[0089] Evaluation Overview: Information on the model being evaluated, overview of the evaluation dataset, and evaluation time.

[0090] Overall Score: Shows the model's average MHJS on the overall evaluation set, as well as the average scores of the three dimensions: FC, LC, and IC.

[0091] Capability dimension analysis: Based on the evaluation system in step S1, the average scores of the model on each sub-category of questions under "professional knowledge ability" and "general ability" are calculated to form radar charts or bar charts to intuitively show the model's strengths and weaknesses.

[0092] Error Case Analysis: List several examples of low scores (severe illusions), including questions, standard answers, model output answers, and analysis of points lost in each dimension.

[0093] Conclusions and Recommendations: This study summarizes the main types and distribution patterns of model illusions, providing specific suggestions for subsequent model fine-tuning, optimization, or application deployment.

[0094] In some embodiments of the present invention, in step S1, the professional knowledge capabilities include military-related capabilities, aerospace-related capabilities, and command invocation capabilities.

[0095] Taking the military aerospace field as an example, the detailed definitions of the three major capability categories are as follows:

[0096] (1) Professional knowledge and skills:

[0097] Military field: can be further subdivided into subcategories such as military reconnaissance and intelligence, military operations theory, military equipment and target identification, geopolitical analysis, organization and troop management, etc.

[0098] Aerospace sector: can be further subdivided into satellite applications (remote sensing, navigation, communication), aerospace equipment (launch vehicles, spacecraft, space stations), etc.

[0099] Command Invocation Class: This class primarily evaluates the model's ability to understand complex user intentions, plan task steps, invoke tools, and execute them accurately when acting as an intelligent agent.

[0100] (2) General abilities:

[0101] World knowledge: Assess the model's grasp of general knowledge such as historical events, geographical common sense, and scientific principles.

[0102] Reasoning ability: Assess the model's logical thinking abilities, such as deductive reasoning, inductive reasoning, and numerical calculation.

[0103] Language ability includes text comprehension (reading comprehension, semantic analysis) and language generation (fluency, grammatical correctness, stylistic consistency).

[0104] (3) Performance:

[0105] Alignment capability: Assess whether the model's response to sensitive topics (such as religion, politics, terrorism), illegal information, or ethical dilemmas complies with security norms.

[0106] Robustness: Evaluates whether the output of a model is stable and reliable when faced with inputs containing noise, disturbances, or adversarial designs.

[0107] Efficiency: Evaluate the model's average response latency, throughput, and computational resource usage during inference.

[0108] In some embodiments of the present invention, such as Figure 3 As shown, the model-generated question-and-answer pairs are scored based on three dimensions: factual consistency, logical consistency, and information completeness, targeting areas where hallucinations are likely to occur.

[0109] Factual consistency score refers to the degree of match between the key information or parameters of the assessment content and the answer, and is calculated using the following formula:

[0110] ,

[0111] in, Score for factual consistency. n represents the number of key information items in the answer. For the factual difference degree of the i-th key information, when the key information is a number or a parameter, When the key information is text, , For key information in the reference answer, This provides key information for the output results.

[0112] The closer the FC score is to 1, the higher the factual consistency; the closer it is to 0, the more severe the illusion (factual error).

[0113] Logical coherence refers to assessing the internal logical coherence of content. It involves scoring the text based on an understanding of its structure and content using a reasoning model, with a scoring range of... .

[0114] The implementation of logical consistency scoring (LC) relies relatively heavily on the comprehension ability of the judges. For example, the assessment prompts may explicitly ask the judges to: "Please assess whether the logic within the 'output answer' is coherent and reasonable, and whether there are any contradictions or inconsistencies. Based on your understanding, please give a score between 0 and 1, where 1 represents complete logical consistency and 0 represents serious logical contradictions."

[0115] The judging models (such as Qwen and GPT) analyze and score the output answers based on this instruction, combined with their own logical reasoning abilities. Although this scoring has a certain degree of subjectivity, because it uses a relatively strong unified judging model and the scoring criteria for the same batch of answers are relatively consistent, its results still have important reference value in horizontal comparisons.

[0116] Information completeness refers to whether the content covers the key information points of the answer, and is calculated using the following formula:

[0117] ,

[0118] in, Scoring for information completeness, , To determine the number of key information points in the reference answer, This represents the number of key information items in the output.

[0119] The closer the IC score is to 1, the more complete the information in the output answer; the lower the score, the more core information is omitted, and the more obvious the illusion of incomplete information.

[0120] The comprehensive model hallucination score MHJS calculation, in addition to the arithmetic mean described above ( ), or a weighted average can be used depending on the assessment requirements:

[0121] ,

[0122] Here, w1, w2, and w3 represent the weights for factual consistency, logical consistency, and information completeness, respectively, and w1 + w2 + w3 = 1. For example, in scenarios emphasizing factual accuracy, w1 = 0.5, w2 = 0.3, and w3 = 0.2 can be set.

[0123] In some embodiments of the present invention, in the "automatic initial screening of the model" in step S3, the text classification model (Fasttext model) needs to be pre-trained. For example, the training and usage process of the Fasttext model is as follows:

[0124] Training data preparation: A certain number of question-answer pairs are randomly selected from the historical generated results and manually labeled as "high quality" or "low quality" according to the "question-answer pair quality evaluation criteria" mentioned above.

[0125] Model training: The Fasttext classification model is trained using labeled data. The Fasttext model is chosen because of its fast training speed, good performance, and suitability for text classification tasks.

[0126] Automatic filtering: All question-answer pairs from the initial evaluation dataset are input into the trained Fasttext model. The model predicts a category label and confidence score for each question-answer pair. Question-answer pairs with confidence scores below a certain threshold or those directly predicted as "low quality" can be automatically filtered out.

[0127] In step S3, “Manual Fine Screening”, the detailed dimensional definitions of the “Question-Answer Pair Quality Evaluation Standard” are as follows (taking the military field as an example):

[0128] Language fluency: Whether the questions and answers are grammatically correct and flow smoothly and naturally.

[0129] Question-answer relevance: Does the answer directly and accurately address the question posed?

[0130] Question and answer validity: Does the question have practical significance and testing value, and does the answer provide effective information?

[0131] Question-answer completeness: For compound questions, does the answer cover all sub-questions?

[0132] Question and answer accuracy: Whether the facts, data, and opinions in the answer are consistent with the context or generally accepted knowledge in the field.

[0133] Format compliance: Whether the required output format (such as JSON) is strictly followed.

[0134] According to one aspect of the present invention, an evaluation system based on the illusion of a large domain model is proposed for implementing the method as described in any of the above technical solutions, comprising:

[0135] The evaluation system construction module is used to divide the capabilities of the domain model to be evaluated into three categories: professional knowledge capabilities, general capabilities, and performance capabilities, and to further define each type of capability in detail to construct an evaluation system for the domain model.

[0136] The dataset construction module is used to collect multi-format document data in the target domain, convert them into a structured text format and slice them; use a large language model to generate domain knowledge question-answer pairs based on the document slices, and perform deduplication on the generated question-answer pairs to form an initial evaluation dataset;

[0137] The data filtering module is used to automatically filter the initial evaluation dataset using a trained text classification model, and to manually filter the dataset based on preset question-and-answer criteria to obtain a high-quality evaluation dataset.

[0138] The illusion assessment module is used to compare the output answer of the large model in the domain to be assessed with the standard answer in the high-quality assessment dataset based on the judge model. The assessment includes: requiring the judge model to extract key information and summarize the structured information of the answer through assessment prompts, and scoring it from three dimensions: factual consistency, logical consistency and information completeness, so as to calculate a comprehensive model illusion judgment score.

[0139] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform an evaluation method based on domain large model illusion as described in any of the above technical solutions.

[0140] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0141] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units. The memory is used to store the computer program and other programs and data required by the terminal device. It can also be used to temporarily store data that has been output or will be output.

[0142] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement an evaluation method based on the illusion of a large domain model as described in any of the above technical solutions.

[0143] For example, computer-readable storage media can be read-only memory (ROM), random access memory (RAM), read-only optical disc (CD-ROM), magnetic tape, floppy disk, and optical data storage devices. They can be implemented using computer-executable program code, thus allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this invention is not limited to any particular hardware and software combination.

[0144] In summary, this invention proposes an evaluation method based on the illusion of a large domain model. It constructs a domain-wide large model evaluation system, comprehensively analyzes model capabilities and evaluation dimensions, and categorizes domain-wide large model capabilities into three main types: professional knowledge capabilities, general capabilities, and performance. Data screening criteria are established, employing a model-human dual-layer screening mechanism to construct a reliable and comprehensive large model evaluation dataset. Through a referee-based model illusion evaluation method, a model evaluation prompt is constructed to extract and comprehensively score the content generated by the model from multiple dimensions. Compared to traditional large model evaluation methods (ROUGE, BLEU, etc.), this method comprehensively analyzes model capabilities and evaluation dimensions to provide a holistic evaluation of model capabilities. When evaluating the model's professional knowledge capabilities, it comprehensively scores the content generated by the model from multiple dimensions, demonstrating high sensitivity to key information generated by the model (core knowledge, concepts, and related parameters). Furthermore, during the construction of the evaluation dataset, the reliability of the evaluation dataset is improved by optimizing the data processing and screening process. This invention solves the problems of inaccurate domain-wide large model illusion evaluation and incomplete and unreliable evaluation dataset generation in existing technologies.

[0145] Furthermore, it should be noted that the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0146] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0149] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. An evaluation method based on the illusion of a large domain model, characterized in that, Includes the following steps: Step S1: Divide the capabilities of the domain model to be evaluated into three categories: professional knowledge capabilities, general capabilities, and performance capabilities, and further define each type of capability in detail to construct a domain model evaluation system. Step S2: Collect multi-format document data in the target domain, and use a large language model with reasoning capabilities to generate domain knowledge question-answer pairs based on the document slices to form an initial evaluation dataset; Step S3: Based on the initial evaluation dataset, a model-human two-layer screening mechanism is used to obtain a high-quality evaluation dataset; Step S4: Compare and evaluate the output answer of the large model in the domain to be evaluated with the standard answer in the high-quality evaluation dataset.

2. The method according to claim 1, characterized in that, In step S1, the professional knowledge capabilities include military-related capabilities, aerospace-related capabilities, and command invocation capabilities; The general abilities mentioned include world knowledge, reasoning ability, and language ability; The performance characteristics include alignment capability, robustness, and efficiency.

3. The method according to claim 2, characterized in that, Step S2 specifically includes: Step S21: Collect document data in the target domain and convert it into Markdown format. Step S22: Based on the chapter or title structure of the Markdown format document, cut the document into multiple semantically complete text slices; Step S23: Input the text slice into a preset large language model to generate knowledge question-answer pairs related to the slice content. The question-answer pairs are stored in JSON format. Step S24: Calculate the text vectors of the generated question-answer pairs, and deduplicate question-answer pairs with similarity exceeding a preset threshold based on vector similarity to form an initial evaluation dataset.

4. The method according to claim 1, characterized in that, Step S3 specifically includes: A trained text classification model is used to automatically filter the question-answer pairs in the initial evaluation dataset, removing low-quality question-answer pairs. Based on the preset question-answer pair quality evaluation criteria, the question-answer pairs selected by the model are manually screened to obtain a high-quality evaluation dataset. The text classification model is the Fasttext model, which is trained on question-and-answer pairs labeled with high-quality and low-quality tags. The preset quality evaluation criteria for the question-and-answer pairs include at least one of the following dimensions: language fluency, question-and-answer relevance, question-and-answer effectiveness, question-and-answer completeness, question-and-answer accuracy, and format standardization.

5. The method according to claim 1, characterized in that, Step S4 specifically includes: The judge model is constructed to evaluate prompt words. The prompt words require the judge model to first extract and structure key information from the output answer and the standard answer respectively, and then score the output answer from three dimensions: factual consistency, logical consistency and information completeness. Based on the scoring results of the three dimensions, a comprehensive model illusion judgment score is calculated.

6. The method according to claim 5, characterized in that, In step S4, the factual consistency score is... Calculated using the following formula: ， in, Score for factual consistency. n represents the number of key information items in the answer. For the factual difference degree of the i-th key information, when the key information is a number or a parameter, When the key information is text, , For key information in the reference answer, This provides key information for the output results.

7. The method according to claim 5, characterized in that, In step S4, the information integrity score is... Calculated using the following formula: ， in, Scoring for information completeness, , To determine the number of key information points in the reference answer, The number of key information items in the output.

8. The method according to claim 5, characterized in that, In step S4, the logical consistency score (LC) is directly scored by the judge model within the range of 0 to 1 based on its understanding of the internal logical coherence of the output answer.

9. The method according to claim 5, characterized in that, In step S4, the model illusion score MHJS is obtained by weighted sum or arithmetic mean of the fact consistency score FC, the logical consistency score LC, and the information integrity score IC.

10. An evaluation system based on the illusion of a large domain model, for implementing the method as described in any one of claims 1 to 9, characterized in that, include: The evaluation system construction module is used to divide the capabilities of the domain model to be evaluated into three categories: professional knowledge capabilities, general capabilities, and performance capabilities, and to further define each type of capability in detail to construct an evaluation system for the domain model. The dataset building module is used to collect multi-format document data in the target domain, convert them into a structured text format and perform document slicing; Using a large language model, domain knowledge question-answer pairs are generated based on the document slices, and the generated question-answer pairs are deduplicated to form an initial evaluation dataset; The data filtering module is used to automatically filter the initial evaluation dataset using a trained text classification model, and to manually filter the dataset based on preset question-and-answer criteria to obtain a high-quality evaluation dataset. The illusion assessment module is used to compare the output answer of the large model in the domain to be assessed with the standard answer in the high-quality assessment dataset based on the judge model. The assessment includes: requiring the judge model to extract key information and summarize the structured information of the answer through assessment prompts, and scoring it from three dimensions: factual consistency, logical consistency and information completeness, so as to calculate a comprehensive model illusion judgment score.

Images