Method and device for evaluating large language model

Abstract

The embodiment of the invention provides a method and device for evaluating a large language model. The method comprises the steps of obtaining an evaluation set, wherein the evaluation set comprises user questions and intention categories corresponding to the user questions; inputting the user question into the large language model to obtain an answer output by the large language model; determining a plurality of to-be-evaluated indexes based on the intention category of the user question; and evaluating the answer based on each to-be-evaluated index to obtain an evaluation result. According to the method, the evaluation result of the model performance is closer to the real feeling of a user.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, specifically to a method and apparatus for evaluating large language models. Background Technology

[0002] Large Language Models (LLMs), often shortened to large models, are deep learning models trained on massive text corpora, containing hundreds of millions or more parameters, and based on natural language processing. Users can interact with these large models to obtain answers to their questions. As the capabilities of large language models continue to improve, they are increasingly used to solve various problems in people's daily lives, making it crucial to accurately measure their performance and reliability.

[0003] Current evaluation methods typically use benchmark sets and uniformly employ fixed metrics (such as accuracy and similarity) to compare the model output with the reference answer for various types of questions. However, real-world user questions are often open-ended, complex, and multi-intent-driven. The aforementioned consistency comparison methods are insufficient to provide evaluation results that reflect the actual user experience, leading to potential discrepancies between evaluation results and actual usage effects. This could result in a high model score, but users may have a poor experience when actually using the large model.

[0004] Therefore, a large-scale model evaluation method is needed to accurately measure whether the model performance truly meets the actual needs of users. Summary of the Invention

[0005] This specification provides a method and apparatus for evaluating large language models, which can make the evaluation results of model performance more closely reflect the user's real experience.

[0006] In a first aspect, embodiments of this specification provide a method for evaluating a large language model, comprising: obtaining an evaluation set, the evaluation set including user questions and intent categories corresponding to the user questions; inputting the user questions into a large language model to obtain an answer output by the large language model; determining several indicators to be evaluated based on the intent categories of the user questions; and evaluating the answer based on each of the indicators to be evaluated to obtain an evaluation result.

[0007] In some optional embodiments, determining several metrics to be evaluated based on the intent category of the user question includes: determining at least one high-level metric to be evaluated based on the intent category of the user question, wherein the high-level metric includes richness, usability, insight, and inspiration; evaluating the answer based on each of the metrics to be evaluated to obtain an evaluation result includes: evaluating the answer based on at least one high-level metric and a basic metric to obtain an evaluation result, wherein the basic metric includes factuality, completeness, and relevance.

[0008] In some optional embodiments, evaluating the answer based on each of the indicators to be evaluated to obtain an evaluation result includes: for each indicator, scoring the answer using the evaluation method corresponding to the indicator to obtain a scoring result, wherein the evaluation method includes at least one of the following: checklist-based evaluation, scoring rule-based evaluation, and fact-checking; and obtaining the evaluation result based on the scoring results of each of the indicators.

[0009] In some optional embodiments, obtaining the evaluation set includes: obtaining user questions and intent categories of the user questions, and constructing an evaluation set based on the user questions and intent categories.

[0010] In some optional embodiments, obtaining the intent category of the user question includes: classifying the user question into intent categories by multiple first-large models to obtain corresponding user intent categories; and determining the user intent category that appears most frequently as the intent category of the user question.

[0011] In some optional embodiments, constructing an evaluation set based on the user question and the intent category includes: generating a checklist containing multiple check items based on the user question and the associated standard answer, according to the checklist generation method corresponding to the type of the user question, and the checklist corresponding to key information points in the user question and / or the standard answer; constructing an evaluation set based on the user question, the intent category, and the checklist, wherein the checklist in the evaluation set is used to check the key information points in the answer.

[0012] In some optional embodiments, the user question is of any of the following types: open-ended question, time-sensitive question, multi-hop reasoning question; wherein the open-ended question is a question without a fixed answer in a real-world scenario, the time-sensitive question is a question that changes over time, and the multi-hop reasoning question is a question that requires multiple intermediate reasoning steps to solve.

[0013] In some optional embodiments, obtaining the user question includes: obtaining news data, generating the timeliness question based on key information points in the news data, the key information points including entities, time, location and data, and the news data containing standard answers associated with the timeliness question.

[0014] In some optional embodiments, the step of generating a checklist containing multiple check items based on the checklist generation method corresponding to the type of user question and the standard answer associated with the user question includes: inputting the timeliness question and the news data into a second large model, and generating a checklist containing multiple check items from the second large model.

[0015] In some optional embodiments, the checklist generation method corresponding to the type of user question generates a checklist containing multiple check items based on the user question and the standard answer associated with the user question: based on the open-ended question, multiple reference answers are generated from several third-party models; based on the multiple reference answers, the standard answer associated with the open-ended question is generated; based on the key information points in the standard answer, check items corresponding to the key information points are generated respectively to obtain the checklist.

[0016] In some optional embodiments, obtaining the user question includes: extracting relation triples based on a first data sample in the knowledge base, the relation triples representing the relationship between a first entity and a second entity; obtaining a second data sample based on the second entity; generating a multi-hop reasoning question based on the first data sample, the second data sample, and the relation triples, the multi-hop reasoning question being used to query based on key information points corresponding to the second entity in the second data sample; and generating a checklist containing multiple check items based on the user question and the associated standard answer, according to the checklist generation method corresponding to the type of the user question, including: generating check items corresponding to each intermediate reasoning step based on multiple intermediate reasoning steps corresponding to the multi-hop reasoning question and the associated standard answer, thus obtaining the checklist.

[0017] In some optional embodiments, obtaining the user question includes: extracting multiple target triples based on the target entity of a third data sample in the knowledge base, wherein the target triples are used to represent the relationship between the target entity and different tail entities; obtaining several search keywords based on the tail entities; obtaining multiple fourth data samples through the search keywords, wherein the fourth data samples include clue information of the tail entities; generating a multi-hop reasoning question based on the clue information of the tail entities and the reasoning relationship between the target entity, wherein the multi-hop reasoning question is used to identify the target entity based on the clue information of the tail entities, and the target entity is the standard answer associated with the multi-hop reasoning question; and generating a checklist containing multiple check items based on the user question and the standard answer associated with the user question, according to the checklist generation method corresponding to the type of the user question, including: generating a checklist containing multiple check items based on the target entity and the tail entities in each target triple.

[0018] In some optional embodiments, each key information point corresponds to several check items, which are used to check whether the key information point exists and / or whether the key information point is accurate.

[0019] Secondly, embodiments of this specification provide an apparatus for evaluating a large language model, comprising: an acquisition module for acquiring an evaluation set, the evaluation set including user questions and intent categories corresponding to the user questions; an inference module for inputting the user questions into the large language model to obtain an answer output by the large language model; a determination module for determining several indicators to be evaluated based on the intent categories of the user questions; and an evaluation module for evaluating the answer based on each of the indicators to be evaluated to obtain an evaluation result.

[0020] Thirdly, embodiments of this specification provide a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the implementations in the first aspect.

[0021] In the above-mentioned scheme provided in this specification, by evaluating the user intent represented by the intent category of the user questions in the evaluation set, the matching evaluation index is dynamically selected, and the answers of the large language model are evaluated differently according to the different user intents. This makes the evaluation results of the model performance closer to the user's real feelings and accurately reflects the actual performance of the model under diverse user needs. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating two methods for generating multi-hop reasoning problems in the embodiments of this specification;

[0024] Figure 2 This is a flowchart of a method for evaluating a large language model in the embodiments of this specification;

[0025] Figure 3 This is a schematic diagram illustrating the mapping relationship between indicators and evaluation methods in the embodiments of this specification;

[0026] Figure 4 This is a schematic diagram of the device for evaluating large language models in the embodiments of this specification. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0028] In this specification, the large language model may also be referred to simply as the large model. A large language model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large language models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to achieve global modeling of sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large language models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, enabling them to generalize. The core capabilities of large language models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Their usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the large language model to generate specific outputs by designing prompts. Prompts can be task descriptions or instructions in text form, used to stimulate the large language model's semantic understanding and generation capabilities. In fine-tuning mode, large language models are further trained on small-scale datasets within a specific domain to optimize their performance on specific tasks. The powerful generalization capabilities and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0029] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0030] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0031] As mentioned earlier, with the continuous improvement of large language models, they are increasingly used to solve various problems in people's daily lives, making it crucial to accurately measure their performance and reliability. Existing evaluations typically use benchmark sets and uniformly employ fixed metrics (such as accuracy and similarity) to compare the model output with the reference answer for various types of questions. However, user questions in real-world applications are often open-ended, complex, and multi-intent-driven. The aforementioned consistency comparison methods struggle to provide evaluation results that reflect the actual user experience, potentially leading to discrepancies between evaluation results and actual usage effects. This could result in a high model score, but users may have a poor experience when actually using the large model, meaning that current evaluation methods are disconnected from the user's real-world experience.

[0032] Based on this, this specification provides a scheme for evaluating large language models to accurately measure whether the model's performance truly meets the user's actual needs. First, an evaluation set is obtained, including user questions and their corresponding intent categories. Then, the user questions are input into the large language model to obtain the model's output answers. Next, based on the intent categories of the user questions, several metrics to be evaluated are determined. Finally, the answers are evaluated based on these metrics to obtain the evaluation results. The method provided in this specification can be used to evaluate various types of large models with question-and-answer functionality. It can dynamically select matching evaluation metrics based on the user intent represented by the intent categories of user questions, and differentiate the evaluation of the large language model's answers according to different user intents. This makes the model performance evaluation results closer to the user's real experience and accurately reflects the model's actual performance under diverse user needs.

[0033] To facilitate understanding of the evaluation scheme in this specification, the construction process of the evaluation set used will be explained first.

[0034] In this embodiment, user questions and intent categories can be obtained, and an evaluation set can be constructed based on user questions and intent categories.

[0035] The intent category of a user question is used to label the user intent within the question. User intent refers to the essential purpose of a user's cognitive goal or information need when asking a question. It reflects the deeper motivation behind the user's question, rather than just the literal meaning of the question. In this embodiment, the intent category of a user question is determined based on the user intent within different user questions, without imposing any restrictions on the specific category classification of user intent.

[0036] For example, user intent can be divided into five categories: Knowledge Retrieval, Knowledge Understanding, Knowledge Application, Knowledge Analysis, and Knowledge Exploration, which correspond to Remember, Understand, Apply, Evaluate, and Create in Bloom's Taxonomy, respectively.

[0037] Specifically, when a user question instructs the user to recall or retrieve a specific fact, definition, or information, the intent category is knowledge retrieval, such as: "How to read a CSV file in Python?"; when a user question instructs the user to explain, summarize, or paraphrase a concept to confirm their understanding, the intent category is knowledge comprehension, such as: "Can you explain quantum entanglement in layman's terms?"; when a user question instructs the user to apply existing knowledge to new situations to solve practical problems, the intent category is knowledge application, such as: "How can I use machine learning to predict stock prices?"; when a user question instructs the user to compare, break down, reason, or evaluate the relationships or logic between different pieces of information, the intent category is knowledge analysis, such as: "Why does the range of new energy vehicles decrease significantly in northern winters? Please analyze from the perspective of battery chemistry and thermal management."; and when a user question instructs the user to engage in creative thinking, propose new viewpoints, envision future possibilities, or conduct open-ended discussions, the intent category is knowledge exploration, such as: "If humans could upload their consciousness, what fundamental changes would occur in social structure?".

[0038] Furthermore, the five types of user intents mentioned above can be further subdivided into 12 sub-intents to more precisely depict the cognitive processes underlying user questions (such as "comparative analysis," "causal inference," and "solution design"). See Table 1 below:

[0039]

[0040] Table 1

[0041] Fact retrieval refers to users directly querying specific facts, data, or information, such as objective information like dates, locations, people, and quantities, requiring accurate answers rather than complex analysis or explanations. Resource retrieval refers to users requesting specific resources, tools, materials, or channels, such as e-book resources, website links, applications, document templates, or learning materials. Information summarization refers to users refining, summarizing, and integrating complex or large amounts of information, extracting key points, identifying patterns, or outlining logical relationships to form a clear and concise overview. Concept explanation refers to users seeking clear explanations of technical terms, abstract concepts, or complex principles to make them easy to understand, including terminology definitions, working principles, and basic knowledge explanations. Solution recommendations refer to users seeking practical solutions to specific problems or difficulties, including troubleshooting, problem fixing, technical implementation methods, or strategies for addressing real-world challenges. Planning refers to users seeking assistance in developing organized and feasible action plans or roadmaps, including timelines, resource allocation, step design, and systematic execution plans. Deductive analysis refers to users using known facts, data, and patterns, employing logical reasoning, data analysis, or causal relationships to deduce subsequent developments or results; or analyzing possible development paths and chain reactions under hypothetical conditions. Evaluative analysis refers to users' desire to conduct multi-dimensional assessments of things, viewpoints, products, or solutions, analyzing their advantages and disadvantages to form an objective and comprehensive evaluation. Value clarification refers to users' desire to conduct in-depth analysis and differentiation of controversial issues or diverse viewpoints, weighing different value orientations and providing balanced and thoughtful insights. Theoretical research refers to users' desire to construct new theoretical models that break through existing cognitive frameworks, proposing new viewpoints or hypotheses based on existing knowledge, exploring unsolved problems, filling theoretical gaps, and promoting conceptual innovation and expanding the boundaries of knowledge. Technological innovation refers to users' desire to propose new methods, technologies, or innovative solutions to address complex challenges, including creative explorations of design concepts, technological paths, or cross-disciplinary integration, characterized by long-term and resource-intensive nature.

[0042] This embodiment does not limit the method of obtaining user questions. For example, user questions can be obtained from existing benchmark datasets, or user questions can be manually constructed. Alternatively, various types of user questions can be collected from real-world sources such as social media platforms, online forums, and other applications.

[0043] This embodiment does not limit the method for obtaining the intent category of user questions. For example, the intent category of user questions can be manually labeled, or a classification model can be used to classify user questions and determine their intent category.

[0044] In one example, a multi-LLM voting mechanism can be employed. Multiple primary models classify the user question's intent, resulting in corresponding user intent categories. The category with the most frequent occurrences is then determined as the user question's intent category. For instance, three different types of primary models can be used. The user question is input into a primary model, which then votes from five user intent categories to select the intent category closest to the question. The category receiving at least two votes is chosen as the final intent category. If a user question lacks consensus, it can be removed. Alternatively, multiple primary models can classify the user question based on 16 sub-intents, resulting in corresponding sub-intent categories. The user intent category belonging to the sub-intent category with the most frequent occurrences is then determined as the user question's intent category, achieving a more accurate classification.

[0045] It is understood that the naming conventions of the first, second, and third major models mentioned in this specification do not limit the type of major model used, but are only for distinguishing the major models used in different stages. The aforementioned major models may be the same major model or different major models.

[0046] Furthermore, to construct a rich and diverse evaluation set, this specification defines different domains for user questions. For example, they can be divided into 16 knowledge domains: Digital Technology, Arts and Entertainment, Culture and Tradition, Sporting Activities, Academic Knowledge, Education and Learning, Health and Medical Care, Business and Finance, Law and Policy, Environment and Nature, Society and Life, Science and Engineering, History and Geography, Food and Beverage, Travel and Transportation, and others. Similarly, a multi-LLM voting mechanism can be used to categorize user questions into domains to determine their domains. When constructing the evaluation set, user questions can be selected from each domain to create a diverse set of evaluation questions.

[0047] Given that existing benchmark datasets primarily consist of closed-ended multiple-choice or short-answer questions, while real-world user questions often involve open-ended tasks such as recommendation and planning, this format difference leads to evaluation results that fail to accurately reflect the model's performance in real-world interactions. In one embodiment, the user questions in the constructed evaluation dataset include open-ended questions. Furthermore, static benchmark datasets have low update frequencies, failing to cover timely content (such as breaking news), and long-term fixed test questions are easily absorbed by LLM training data, resulting in distorted evaluation results (i.e., data pollution). In one embodiment, the user questions in the constructed evaluation dataset include timely questions. Additionally, traditional benchmark datasets focus on isolated knowledge memorization or simple reasoning abilities, neglecting higher-order abilities required in real-world problems, such as multi-step retrieval and cross-domain knowledge integration. For example, analyzing the impact of news policies requires combining legal interpretation with economic reasoning. In one embodiment, the user questions in the constructed evaluation dataset include multi-step reasoning questions.

[0048] Among them, open-ended questions are real-world scenarios without fixed answers, time-sensitive questions are questions that change over time, and multi-hop reasoning questions are questions that require multiple intermediate reasoning steps to solve. Open-ended questions are used to evaluate the model's comprehensive capabilities in real-world scenarios, time-sensitive questions are used to evaluate the model's timeliness based on regularly updated news content, and multi-hop reasoning questions are used to test the model's deep retrieval and reasoning capabilities in complex knowledge networks.

[0049] Understandably, closed-ended and open-ended questions are concepts used to describe question types and answer formats. Their core difference lies in the certainty and freedom of the answer. Closed-ended questions have pre-defined answer options, or the answer itself is very short and specific, and usually only one objectively correct answer. Common forms include multiple-choice questions, true / false questions, or short-answer questions. Short-answer questions have very short and unique answers; for example, "What is the chemical formula of water?" The correct answer can only be "H2O." Open-ended questions, on the other hand, offer diverse and free answers, without a single pre-defined answer. They place greater emphasis on the quality, depth, and logic of the response, and are difficult to automate. They cannot be simply judged by comparing text; a more complex evaluation framework is needed to measure the quality of the response across multiple dimensions. They are also closer to real-world scenarios; most questions people ask AI assistants in daily life are open-ended, such as seeking advice, planning, explanations, or creative work. Common forms of open-ended questions include: essay / explanation questions, such as explaining how inflation affects the lives of ordinary people; suggestion / planning questions, such as planning a three-day trip to Beijing next week; creative / generating questions, such as writing a poem about spring; and comparison / analysis questions, such as comparing the advantages and disadvantages of electric vehicles and gasoline-powered vehicles.

[0050] In some optional embodiments, to facilitate subsequent evaluation processes that assess user questions based on checklists, a checklist can be set for each user question when constructing the evaluation set. Specifically, based on the checklist generation method corresponding to the type of user question, a checklist containing multiple check items can be generated based on the user question and its associated standard answer. Then, the evaluation set is constructed based on the user question, intent category, and checklist.

[0051] The checklist in the evaluation set is used to examine key information points in the answers, such as assessing the completeness and factual accuracy of the answers. A checklist can be a list of multiple specific and verifiable checks. These checks correspond to key information points in the user question and / or the standard answer. These checks can be designed as closed-ended questions, such as true / false questions to determine "yes / no," or scoring questions with rating levels. For example, a checklist could be "Is the year 2008?"

[0052] For example, a check item can use the following template: "Whether + [Verb] + [Checkpoint] + [Verb] + [Fact Point]". Here, "Whether" is the introductory word of the entire sentence, transforming a declarative sentence into a closed-ended "yes / no" question, making the check item a binary multiple-choice or scoring question that can be directly judged as right or wrong, facilitating scoring in the subsequent evaluation process. The first [verb] describes the action or state that the model needs to perform in its response, explicitly requiring the model's response to include certain behaviors. For example, "mentioned" is more specific than "said," emphasizing the existence of information; "explained" requires an explanation process. The [checkpoint] is the core content of the check item, i.e., the key information point or concept that needs to be verified. It usually comes from the core elements of the user's question or standard answer. Each check item should focus on only one key information point. For example, for "Who is the third enemy in 'The Biography of XX'?", the [checkpoint] is "The third enemy in 'The Biography of XX'". The second verb serves as a bridge connecting the checkpoint and the fact point. Common verbs include: is, is, involves, belongs to, and is related to. Its purpose is to establish the relationship between the checkpoint and the fact point, indicating that the previously mentioned checkpoint must be associated with or match the specific fact point that follows. The fact point is the specific, objective fact that ultimately needs to be verified. It comes from verified and confirmed facts in the standard answer. For example, for "Who is the third enemy in 'The Biography of XX'?", the fact point is "A" (the name of the third enemy).

[0053] In some optional embodiments, each key information point corresponds to several check items, which are used to check whether the key information point exists and / or whether the key information point is accurate.

[0054] For example, for each key information point, a check item can be generated to verify the existence of the key information, thus evaluating the completeness indicator. This check item could be: "Does the key information point exist?", for example, "Does it include year information?". Similarly, for each key information point, a check item can be generated to verify the accuracy of the key information, thus evaluating the factuality indicator. This check item could be: "Are the key information points in the answer the preset values?", for example, "Is the year 2008?". Furthermore, for each key information point, a pair of check items can be generated, one to evaluate factuality and the other completeness.

[0055] In this embodiment, different checklists can be generated for different types of user questions. For example, for open-ended questions, the checklist mainly covers the completeness of the answer dimension; for time-sensitive questions, the checklist mainly checks the completeness and factual accuracy of time information; and for multi-hop reasoning questions, the checklist covers the completeness and factual accuracy of each reasoning step and reasoning result.

[0056] The following three examples illustrate how to obtain the three types of user questions and how to generate checklists.

[0057] Example 1: Open-ended questions

[0058] For open-ended questions, we can first collect various user questions from user queries, discussions, and comments on popular real-world websites and applications, while simultaneously using AI to filter out low-quality questions. For example, LLM can be used to filter out low-quality questions based on the following principles to achieve user question quality control: Principle 1) Clear topic, filtering out overly vague questions; 2) Clear intent, filtering out questions where the user's intent is unclear; 3) Exclude questions related to code generation, identity verification, and security risks.

[0059] Then, based on the open-ended question, several third-major models can generate multiple reference answers, and based on the multiple reference answers, standard answers associated with the open-ended question can be generated. Then, based on the key information points in the standard answers, check items corresponding to the key information points are generated to obtain a checklist.

[0060] It's understandable that open-ended questions don't inherently have a single, fixed answer. The standard answer generated in this step serves as the basis for generating the checklist, not as the sole answer to the user's question. For example, multiple large models with RAG (Retrieval-Augmented Generation) capabilities can be used to generate multiple reference answers, which can then be integrated to generate the standard answer. During the integration process, consensus content from different reference answers can be retained, complementary information can be integrated, and finally, redundant information from multiple reference answers can be eliminated to ensure the completeness and rigor of the standard answer.

[0061] Subsequently, a checklist can be generated by identifying key information points and factual claims in the standard answer. Each checklist item can independently verify a key information point; checklist items should be clearly stated and free of any factual errors. Checklist items that do not meet the requirements can be manually revised. For example, an open-ended question could be: Develop a six-month fitness training plan for a mountaineering enthusiast who often gives up halfway due to insufficient physical strength. Based on the template of the checklist items in the example above, a possible generated checklist includes the following five checklist items for completeness indicators:

[0062] Is a six-month phased training plan provided? (Key information: six-month phased training plan)

[0063] Has the weekly training frequency been specified? (Key information point: weekly training frequency)

[0064] Does it include specific training exercise suggestions for cardiorespiratory endurance? (Key information point: cardiorespiratory endurance)

[0065] Did it mention how to gradually increase training intensity to avoid injury? (Key information: Gradually increase training intensity)

[0066] Did you provide any dietary advice? (Key information point: dietary advice)

[0067] Example 2: Timeliness Issues

[0068] For time-sensitive questions, news data can be obtained from various news reports and policy documents. Time-sensitive questions can be generated based on key information points within this data, including entities, time, location, and data. The news data also contains standard answers that relate to the time-sensitive questions. For example, based on key information points such as entities, time, location, and data, realistic questions that closely match user responses can be generated. Specifically, key information points, such as entities, time or location information, and data information, can be extracted from news content centered around news headlines. Time-sensitive questions can then be generated around these key information points, ensuring that each time-sensitive question has a unique and clear standard answer within the news content.

[0069] Then, the timeliness question and news data are input into the second main model, which generates a checklist containing multiple checks. The news data can be the original news article, containing the answer to the question and all relevant background information, data, and details. The second main model can automatically extract key information points related to the timeliness question and its associated standard answer from the news content and transform them into structured, verifiable checks. For example, it can generate checks for each key information point according to the checklist template mentioned above. For example, for the timeliness question: "When will the newly released environmental protection policy take effect?", the model needs to find the following key information points from the news: policy name (e.g., "Several Measures on Strengthening Air Pollution Prevention and Control"), effective date (e.g., "Effective from June 1, 2025"), issuing agency (e.g., "Ministry of Ecology and Environment"), core measures (e.g., "Banning the burning of straw in urban centers"). The generated checklist can then include the following checks, where for each key information point, a pair of checks for factual accuracy and completeness are generated:

[0070] Did you mention the name of the new policy?

[0071] Is the new policy titled "Several Measures to Strengthen Air Pollution Prevention and Control"?

[0072] Has the effective date of the policy been specified?

[0073] Is the policy effective on June 1, 2025?

[0074] Did you specify the agency that issued the policy?

[0075] Is the Ministry of Ecology and Environment the issuing agency for this policy?

[0076] Does it include the core measures of the policy?

[0077] Is the core measure of the policy to prohibit the burning of straw in urban centers?

[0078] Example 3: Multi-hop reasoning problem

[0079] Multi-hop reasoning problems typically involve reasoning based on vague information provided by the user. This embodiment illustrates two methods for obtaining multi-hop reasoning problems and generating checklists: a multi-hop generation method and a needle-in-a-haystack generation method. The needle-in-a-haystack generation method is suitable for highly abstract scenarios with coarse user questions. It requires gradually narrowing down the scope using contextual or geographical information to locate the core question, such as "Is it worth buying a house recently?" This requires step-by-step communication with the user to obtain auxiliary information such as the user's house location, budget, and needs. The multi-hop generation method is more suitable for systematic and complex problems that require multi-step decomposition, such as analyzing a report.

[0080] The following is combined Figure 1 The multi-hop generation method and the needle-in-a-haystack generation method are explained separately.

[0081] For the multi-hop generation method, firstly, based on the first data sample in the knowledge base, relation triples can be extracted. The relation triples are used to represent the relationship between the first entity and the second entity. Then, based on the second entity, a second data sample is obtained. Next, based on the first data sample, the second data sample, and the relation triples, a multi-hop reasoning question is generated. The multi-hop reasoning question is used to query based on the key information points corresponding to the second entity in the second data sample.

[0082] The knowledge base can be various types of online resources or domain-specific knowledge bases. Data samples can be various types of data files. For example... Figure 1 As shown, LLM can be used to extract relation triples (E1, R1, E2) from a high-quality first data sample (i.e., document Doc1), where E1, R1, and E2 represent entity 1 (i.e., the first entity), relation 1, and entity 2 (i.e., the second entity), respectively. For example, from the TV series document "Ren Changxia" (document Doc1) on XX Encyclopedia, the relation triples (Ren Changxia, Director, Shen Haofang) can be extracted. Then, a search tool can be used to obtain a second data sample (document Doc2) about the second entity E2, for example, Shen Haofang's XX Encyclopedia entry, where the key information for E2 could be the personal introduction section of XX Encyclopedia. Next, the key information and triples related to E2 from Doc1 and Doc2 are input into LLM. When generating questions, the name of E2 can be avoided by explicitly mentioning it, but rather indirectly mentioned through references to E1 and R1 to form a multi-hop relation. The generated questions focus on the key information points of E2 contained in Doc2. For example, the generated multi-hop reasoning question could be "Which movie was released in 2004 by the director who directed the TV series 'Ren Changxia'?" This multi-hop reasoning question is used to query based on the key information point "No. 5 Courtyard" corresponding to the second entity in the second data sample.

[0083] Then, based on the multiple intermediate reasoning steps corresponding to the multi-hop reasoning problem and the standard answers associated with the intermediate reasoning steps, check items corresponding to each intermediate reasoning step are generated to obtain a checklist.

[0084] Intermediate reasoning steps refer to the series of logical deductions a model needs to perform from known information to the final answer when solving a complex multi-hop reasoning problem. Simply put, it's about completing each key link or "jump" in the entire reasoning chain. Because the answer to a problem cannot be directly derived from a single fact, reasoning must proceed step by step. A multi-hop reasoning problem typically involves two or more different sources of information (such as Doc1 and Doc2 in a document), and the answer requires connecting these disparate pieces of information. For example, to answer the question, "Which film released in 2004 was directed by the director of 'Ren Changxia'?", one cannot simply look at the introduction to "Ren Changxia" (Doc1); one also needs to know who the director is, then look up the director's other works (Doc2), and finally find the 2004 film. This requires at least two steps (or "two jumps") of reasoning.

[0085] When generating a checklist for multi-hop reasoning questions generated using a multi-hop generation method, the checklist can be generated for the key information points required for each jump (i.e., intermediate reasoning step) in the question-and-answer process. A jump refers to the process of moving from one information point to another. In the example above:

[0086] First jump (Hop 1): Starting from the TV series "Ren Changxia" (entity E1), jump to director Shen Haofang (entity E2) through the relationship of "director" (R1). This step is based on the information in Doc1.

[0087] The second jump (Hop 2): Starting with director Shen Haofang (entity E2), jump to all his film works (entity E3) through the relationship of "directed films" (R2). This step is based on the information in Doc2.

[0088] The third step (Hop 3): From all of Shen Haofang's films, filter by the "year of release" attribute to find the film released in 2004 ("Courtyard No. 5"). This step combines information from Doc2 with simple comparison operations.

[0089] The standard answer associated with each jump comes from Doc1 or Doc2. For example, the multiple intermediate reasoning steps and associated standard answers for the above multi-hop reasoning problem can be as follows:

[0090] Step 1: Based on Doc1, we can infer that the director of "Ren Changxia" is Shen Haofang.

[0091] Step 2: Extract the films directed by Shen Haofang and their release years from Doc2.

[0092] Step 3: Compare the years to determine that the movie released in 2004 is "No. 5 Courtyard".

[0093] When generating the checklist, check items are created for each intermediate reasoning step and the standard answer. This allows for a precise assessment of whether the model can successfully complete each step of reasoning, thus comprehensively measuring its complex reasoning capabilities. Specifically, the checklist's check items cover key information points in multiple intermediate reasoning steps and associated standard answers to evaluate whether the AI ​​application can establish a complete multi-hop reasoning chain.

[0094] For the "needle in a haystack" generation method, firstly, multiple target triples can be extracted from the target entity in the third data sample of the knowledge base. The target triples are used to represent the relationship between the target entity and different tail entities. Then, based on the tail entities, several search keywords are obtained. Next, multiple fourth data samples are obtained through the search keywords. The fourth data samples include clue information of the tail entities. Finally, based on the clue information of the tail entities and the reasoning relationship between the target entities, a multi-hop reasoning question is generated. The multi-hop reasoning question is used to identify the target entity based on the clue information of the tail entities. The target entity is the standard answer associated with the multi-hop reasoning question.

[0095] like Figure 1 As shown, LLM can be used to specify the target entity Ec (as the standard answer) from the high-quality document Doc1 (i.e., the third data sample) and extract the target triple: [{(Ec, Ri, Ei)} | i ∈ [1, n]]. Here, the tail entity Ei can be a specific person, place, event, competition, or literary work, and the relationships between tail entities are semantically independent. For example, from the third data sample "She Has No Name" (Doc1 document), multiple target triples are extracted: {("She Has No Name", Director, B), ("She Has No Name", Actress, C), ("She Has No Name", Selected, 77th Cannes International Film Festival)}. Then, relevant descriptions of the tail entities are extracted from Doc1, and 2-3 search keywords (e.g., "Director B") are generated. Using these search keywords, multiple fourth data samples (Doc2-Docn) related to the tail entities are retrieved, and clue information of the tail entities is extracted to support the inference relationship in the following multi-hop inference chain: tail entity clue information → tail entity → relationship → target entity. The clues for the tail entity are a unique, specific and verifiable set of descriptive features, which replace directly stating the name of the tail entity, thereby increasing the difficulty of the question and the depth of reasoning. For example, the clue for C is "graduated from the Performance Department of the Central Academy of Drama and was nominated for the Academy Award for Best Actress for Crouching Tiger, Hidden Dragon".

[0096] Then, the multiple logical reasoning chains from the tail entity's clues to the target entity are integrated. Without directly mentioning the tail entity's name, the target entity is inferred from the tail entity's clues and the reasoning relationship between the tail entity and the target entity. For example, the generated multi-hop reasoning problem could be: A Chinese-language film premiered at a top international film festival in 2024. Its lead actress is known for her nuanced and powerful acting, having won an Academy Award for Best Supporting Actress for *Crouching Tiger, Hidden Dragon* and starring in *The Grandmaster*. The film's director was born in Hong Kong in 1962 and is known for depicting the fates of ordinary people amidst social change; his representative work includes *Comrades: Almost a Love Story*. What is the name of this film?

[0097] The multi-hop reasoning problem described above conceals a tail entity; the names "B" or "C" are not directly stated. Instead, a detailed and unique feature description is used, serving as a clue to the tail entity. The corresponding multi-hop reasoning steps for this problem are as follows:

[0098] Step 1 (Starting Point): Identify clues for the "Lead Actress" role → "Graduated from the Central Academy of Drama's Acting Department, won the Academy Award for Best Supporting Actress for *Crouching Tiger, Hidden Dragon*, and starred in *The Grandmaster*." This points to actress C. Identify clues for the director role → "The director of this film was born in Hong Kong in 1962 and is skilled at depicting the fate of ordinary people amidst social changes; his representative works include *Comrades: Almost a Love Story*." This points to director B.

[0099] Step 2 (First Jump): Use the identity "C" to connect to the movies she starred in. This is a jump from "tail entity (C)" to "relationship (starring)". Use the identity "B" to connect to the movies he directed. This is a jump from "tail entity (B)" to "relationship (director)".

[0100] Step 3 (Second Jump): From "Films starring C" and "Films directed by B", select the film that meets the criteria—"Premiered at a top international film festival in 2024". This ultimately points to the target entity, *She Has No Name*.

[0101] Next, a checklist containing multiple checks can be generated based on the target entity and the tail entities in each target triple. The key information points corresponding to the checklist items cover the standard answer and the tail entities in each target triple.

[0102] After generating user questions, intent categories, and checklists through the automated workflow described above, a second quality check can be performed manually to ensure that the generated questions are reasonable, the intent categories are accurately labeled, and the generated checklists are truly relevant to the questions, so as to accurately judge the quality of the model's answer to the questions.

[0103] In the above embodiment, the entire evaluation set construction is divided into four parts: question generation, intent category labeling, checklist generation, and manual review. The entire process is highly automated, and the addition of manual review improves efficiency while ensuring that the evaluation set continues to meet the real needs of users.

[0104] The following section provides a detailed explanation of the method for evaluating large language models provided in the embodiments of this specification, based on the evaluation set constructed above. Figure 2 This is a flowchart of a method for evaluating a large language model in an embodiment of this specification, including steps 201-204 as shown below.

[0105] In step 201, the evaluation set is obtained.

[0106] The evaluation set can include user questions and the intent categories corresponding to the user questions, and may also include checklists.

[0107] In this step, the evaluation set can be obtained by any of the methods constructed in the above embodiments. The method of obtaining the evaluation set will not be described again here.

[0108] In step 202, the user's question is input into the large language model, and the answer output by the large language model is obtained.

[0109] The large language model in this step is the large model to be evaluated. This embodiment can be applied to various types of large language model evaluations. The user's question is input into the large language model to be evaluated. The large language model performs reasoning and outputs an answer to the user's question.

[0110] In step 203, based on the intent category of the user's question, several metrics to be evaluated are determined.

[0111] Each user question in the evaluation set is labeled with its intent category. For example, the intent category can be any of the aforementioned knowledge retrieval, knowledge understanding, knowledge application, knowledge analysis, and knowledge exploration. Based on the pre-established mapping relationship between intent categories and indicators, several indicators to be evaluated can be determined.

[0112] This embodiment does not limit the indicators used and can be set according to actual needs. For example, indicators such as relevance, accuracy, similarity, logic, and interest can be used. Considering that an answer may excel in some aspects but be lacking in others, this embodiment designs a hierarchical and multi-dimensional indicator system. This system includes two layers of indicators: basic indicators and advanced indicators. Basic indicators are fundamental capability indicators, representing the basic requirements that an answer must meet. They are the core indicators that must be evaluated for all questions and measure the basic quality of the answer. They focus on dimensions such as factuality, completeness, and relevance. Advanced indicators, on the other hand, are dynamically selected based on the user's intent regarding the question and are used to evaluate higher-order cognitive abilities, such as richness, practicality, insight, and inspiration.

[0113] The core metrics include factual accuracy, completeness, and relevance. Factual accuracy measures the degree to which the information provided by the model's response conforms to objective facts. Completeness measures the extent to which the model's response comprehensively covers the information required by the question. Relevance measures the degree to which the model's response corresponds to the semantic intent of the question. Advanced metrics include richness, usability, insight, and inspiration. Richness measures the comprehensiveness and breadth of the model's response. Inspiration measures the model's response's ability to stimulate deeper thinking and expand users' understanding. Usability measures the model's response's operability and practical application value. Insight measures the model's response's deep understanding and insight into the essence of the question.

[0114] In this embodiment, instead of using a one-size-fits-all approach with fixed metrics to evaluate LLM, the metrics are dynamically adjusted based on the specific intent of the user's question. From the user's perspective, the gold standard for a truly useful answer adaptively depends on the user's underlying purpose. Therefore, in this embodiment, basic metrics can be evaluated for all questions of different intent categories, while advanced metrics are selected based on the mapping relationship between intent categories and metrics. In this step, at least one advanced metric to be evaluated can be determined based on the intent category of the user's question, so that the answer can be evaluated in subsequent steps based on at least one advanced metric and basic metrics. Table 2 below shows a mapping relationship between intent categories and advanced metrics to be evaluated.

[0115]

[0116] Table 2

[0117] In step 204, the answers are evaluated based on the various indicators to be evaluated, and the evaluation results are obtained.

[0118] In this embodiment, the same evaluation method or different evaluation methods can be used for each indicator. This embodiment does not limit the specific evaluation method; for example, it can be a large-scale model scoring, a score based on preset rules, etc. In one implementation, for each indicator, the answer can be scored using the corresponding evaluation method to obtain a score result. The evaluation method includes at least one of the following: checklist-based evaluation, score rule-based evaluation, and fact-checking; then, based on the score results of each indicator, the evaluation result is obtained. In this embodiment, a large-scale evaluation model with evaluation functions can be used to score the answer based on different evaluation methods.

[0119] like Figure 3 The diagram illustrates the mapping relationship between different metrics and evaluation methods. In other embodiments, other mapping relationships can be set according to actual needs. The various evaluation methods are described below.

[0120] Evaluation methods based on checklists

[0121] This embodiment evaluates factuality and completeness indicators based on a checklist. The checklist is defined as a series of specific and verifiable checks, which can be formulated in the form of closed-ended questions. For a detailed explanation of the checklist items, please refer to the previous embodiment; it will not be repeated here. Each checklist item can be a binary question (yes / no). A large-scale evaluation model with judgment capabilities can be used to evaluate the responses based on the checklist to determine whether the responses meet the checklist requirements.

[0122] Given that binary checks assume the model's answer is either completely satisfactory or completely unsatisfactory, but most checklist items cannot be judged so rigorously and accurately. For example, an answer might be partially correct, have minor errors, or cover most points but omit details. In such cases, binary judgments are too simplistic to accurately evaluate the performance of large models. Therefore, in one embodiment, traditional binary judgments can be extended to a multi-level scoring system based on Likert scales. For example, a 3-point scale (1, 3, and 5 representing completely wrong, partially correct, and completely correct, respectively) can be used for factual indicators, and a 5-point scale (1-5) can be used for completeness indicators, thus providing more granular scores for each checklist item and improving the accuracy of the evaluation results. Furthermore, to save resources, the evaluation model can be prompted to evaluate all checklist items for a given indicator in a single inference process, rather than evaluating only one checklist item at a time.

[0123] Evaluation based on scoring rules (also known as scoring criteria)

[0124] Rule-based assessment is a method for evaluating more complex and subjective metrics that are difficult to measure using simple "yes / no" answers or fixed lists. It breaks down a macro-level evaluation dimension into multiple specific, actionable atomic dimensions by pre-setting a detailed set of scoring criteria, and then scores each atomic dimension independently.

[0125] In this embodiment, a scoring rule-based evaluation method can be applied to the assessment of relevance indicators among the basic indicators and all advanced indicators. Unlike factual and completeness indicators, which can be efficiently decomposed into a series of fixed and objective checklists, the assessment of relevance indicators and other advanced indicators (such as richness) is highly holistic and dependent. Predefining an exhaustive checklist for these indicators is not only extremely difficult but also severely limits the flexibility of the assessment. Therefore, this embodiment employs a scoring rule-based approach, prompting the large evaluation model to assess these indicators according to specific scoring criteria in the atomic dimensions, rather than a mechanical checklist.

[0126] The design of fine-grained atomic dimensions aims to systematically deconstruct the core meaning of each metric. Specifically, in this embodiment, relevance metrics and all advanced metrics predefine fine-grained atomic dimensions, representing different aspects of the metrics to be evaluated. For example, relevance metrics can be divided into two dimensions: core intent responsiveness and extended content redundancy. Furthermore, for those intent-specific advanced metrics, the definition of the corresponding atomic dimensions will differ depending on the question intent being evaluated.

[0127] Specifically, for relevance metrics, they are evaluated using two atomic dimensions: core intent responsiveness, which assesses whether the answer directly addresses the user's core question; and extended content redundancy, which assesses whether the answer contains redundant information unrelated to the core intent. For richness metrics, the general dimensions are: knowledge extensibility, dimensional diversity, and hierarchical progression. Atomic dimensions can be further refined based on different intent categories: for knowledge understanding, the richness of example content can be evaluated; for knowledge application, the richness of implementation details can be evaluated; for knowledge analysis, the richness of viewpoints and evidence can be evaluated; and for knowledge exploration, the richness of innovative thinking can be evaluated. For usability metrics, they are subdivided into three atomic dimensions: clarity of operational instructions; completeness of implementation support; and applicability to real-world scenarios. For insight metrics, they are subdivided into three atomic dimensions: understanding of fundamental mechanisms; unique perspectives; and rigor of the argumentation process. For inspirational metrics, they are subdivided into three atomic dimensions: the degree to which cognitive boundaries are expanded; the ability to guide innovative thinking; and the breadth of exploratory thinking.

[0128] For each atomic dimension, specific fine-grained scoring criteria can be set. For example, using a 5-point scoring system, generally 5 corresponds to excellent, 4 to good, 3 to average, 2 to poor, and 1 to very poor. For instance, the scoring criteria for the atomic dimension "knowledge extensibility" in the richness index could be as follows:

[0129] 5 points: It systematically supplements a large amount of high-value extended information (such as relevant background, development context, and cutting-edge trends), and constructs a complete knowledge network.

[0130] 4 points: Actively provided a considerable amount of high-quality extended information (such as practical cases and important knowledge points), demonstrating professional standards.

[0131] 3 points: Provides some relevant background information or extended knowledge, which has some value and depth.

[0132] 2 points: Limited relevant information provided, with scarce and average extended content.

[0133] 1 point: Almost no relevant information was provided, only basic and direct answers.

[0134] Fact Check

[0135] Considering the two core limitations of checklist-based methods in evaluating factual metrics—that they are impractical for highly open-ended questions and fail to capture wild factual claims outside the scope of predefined checklists—this embodiment employs fact-checking methods as a supplementary evaluation approach for factual metrics.

[0136] For closed-ended questions in knowledge retrieval or understanding, where objective facts can be predefined, factual metrics are evaluated using checklist-based and fact-checking methods. However, for another set of questions (including some open-ended questions in knowledge retrieval and understanding, as well as all questions in knowledge application, knowledge analysis, and knowledge exploration), the factual score comes solely from the fact-checking method. The fact-checking method comprises four main steps: claim extraction → query rewriting → query search → claim evaluation. First, prompts can be used to guide a high-performance evaluation model to extract as many verifiable, objective factual claims as possible from the model's responses, exceeding the scope of the predefined checklist. Then, each claim is rewritten as a valid and concise query. Relevant reference documents are retrieved for each query through web search. Finally, the evaluation model evaluates the correctness of each factual claim based on the reference documents. The scoring criteria can be the same as the factual metrics used in the checklist-based method.

[0137] In other words, for factual indicators, the final score can be calculated by weighting the scores obtained from checklist assessments and fact-checking.

[0138] By summarizing the scores obtained from evaluating each indicator, a comprehensive score is finally calculated as the evaluation result. This embodiment does not restrict the method of score calculation. For example, the calculation process consists of the following two layers:

[0139] The first level: a first-level score can be calculated for each specific evaluation dimension (such as a single check item in a checklist, or an atomic dimension in a scoring rule).

[0140] For example, for checklist-based methods (used for factual and completeness metrics), if a Likert scale (e.g., a 5-point scale) is used, each of the n checklist items is scored individually, and then the average score is calculated: S_metric = (s1 + s2 + ... + s...) n Then, convert the average score to a percentage: S_metric_percentile = ((S_metric -1) / 4) * 100.

[0141] For example, for scoring-based methods (used for relevance metrics and advanced metrics), each of the m atomic dimensions can be scored (from 1 to 5), and then the average score can be calculated: S_metric = (s1 + s2 + ... + s...) m Then convert the average score to a percentage: S_metric_percentile = ((S_metric - 1) / 4) * 100.

[0142] For example, for fact-checking methods (used for factual indicators), k extracted factual claims can be independently judged (correct / incorrect / partially correct), and then their percentage scores can be calculated: S_Fact-Checking = (number of correct claims / total number of claims) * 100.

[0143] The second level: Aggregating multiple first-level scores to obtain scores for more macroscopic "basic indicators" and "advanced indicators", that is, calculating second-level scores.

[0144] Calculate the scores for the basic indicators, which include: factuality, completeness, and relevance, as shown in the formula below. Then, calculate a weighted average of the scores for each basic indicator:

[0145] S_Basic = w2·S_Fact + (1−w2)· (nComp / (nComp + nRel)·SComp + nRel / (nComp + nRel) · SRel) (1)

[0146] Where S_Fact is the factual score (integrated with checklist-based approach and fact-checking), SComp is the completeness score, SRel is the relevance score, w2 is the weighting coefficient, for example, w2 can be set to 05, and (nComp / (nComp + nRel)) and (nRel / (nComp + nRel)) are weighting coefficients based on the proportion of the two indicators in the total check items.

[0147] Additionally, calculate the advanced indicator score, which can be a simple arithmetic average of multiple advanced indicators (such as richness, usability, insight, etc.). The formula is as follows:

[0148] S_Advanced = (S_Rich + S_Prac + S_Insi + S_Insp) / m (2)

[0149] Where S_Rich, S_Prac, S_Insi, and S_Insp are the scores for richness, usefulness, insight, and inspiration, respectively, and m is the total number of high-level indicators evaluated. For example, if four high-level indicators are evaluated, then m=4.

[0150] The third layer calculates the score for each user's question answer, combining the scores of basic and advanced metrics to obtain the total score for each user's question answer. For example, the score for each user's question answer is the geometric mean of its basic and advanced metric scores. This means that if the model performs extremely poorly in one aspect (e.g., very low basic metrics), even if the advanced metrics are high, the final score will be significantly lowered. This emphasizes that answers must reach a certain level in both basic quality and higher-order value.

[0151] Fourth layer: Calculate the score for the entire evaluation set.

[0152] The scores of all user questions are summed to obtain the evaluation result for the entire evaluation set. For example, this can be done by adding up the scores of all user questions, or by averaging or performing an arithmetic score on the scores of each user question.

[0153] Figure 4 This is a schematic diagram of the device for evaluating large language models in the embodiments of this specification. This device can be applied to any device, platform, or cluster of devices with computing and processing capabilities. The device includes:

[0154] Module 41 is used to acquire an evaluation set, which includes user questions and the intent categories corresponding to the user questions;

[0155] The reasoning module 42 is used to input user questions into the large language model and obtain the answer output by the large language model;

[0156] Module 43 is used to determine several metrics to be evaluated based on the intent category of the user's question.

[0157] Evaluation module 44 is used to evaluate the answers based on the various indicators to be evaluated and obtain the evaluation results.

[0158] In some alternative embodiments, the determining module 43 is specifically used to determine at least one high-level metric to be evaluated based on the intent category of the user question. The high-level metric includes richness, usability, insight, and heuristics.

[0159] Evaluation module 44 is specifically used to evaluate the response based on at least one advanced indicator and basic indicator to obtain the evaluation result. The basic indicator includes factuality, completeness and relevance.

[0160] In some optional embodiments, the evaluation module 44 is specifically used to score the answer for each indicator using the evaluation method corresponding to the indicator, and obtain a scoring result. The evaluation method includes at least one of the following: evaluation based on a checklist, evaluation based on scoring rules and fact-checking; and obtaining an evaluation result based on the scoring results of each indicator.

[0161] In some optional embodiments, the acquisition module 41 is specifically used to acquire user questions and intent categories of user questions, and to construct an evaluation set based on user questions and intent categories.

[0162] In some optional embodiments, the acquisition module 41 is specifically used to classify the user question into intents by multiple first major models to obtain the corresponding user intent categories; and to determine the user intent category that appears most frequently as the intent category of the user question.

[0163] In some optional embodiments, the acquisition module 41 is specifically used to generate a checklist containing multiple check items based on the checklist generation method corresponding to the type of user question, based on the user question and the standard answer associated with the user question, the checklist items corresponding to key information points in the user question and / or the standard answer; and to construct an evaluation set based on the user question, intent category and checklist, the checklist in the evaluation set being used to check the key information points in the answer.

[0164] In some optional embodiments, the user question type is any of the following: open-ended question, time-sensitive question, multi-hop reasoning question; wherein, open-ended question is a question without a fixed answer in a real-world scenario, time-sensitive question is a question that changes over time, and multi-hop reasoning question is a question that requires multiple intermediate reasoning steps to solve.

[0165] In some optional embodiments, the acquisition module 41 is specifically used to acquire news data, generate timeliness questions based on key information points in the news data, the key information points including entities, time, location and data, and standard answers in the news data that are associated with timeliness questions.

[0166] In some optional embodiments, the acquisition module 41 is specifically used to input timeliness issues and news data into the second large model, which generates a checklist containing multiple check items.

[0167] In some optional embodiments, the acquisition module 41 is specifically used to generate multiple reference answers from several third-party models based on the open-ended question; generate standard answers associated with the open-ended question based on the multiple reference answers; and generate check items corresponding to the key information points in the standard answers to obtain a checklist.

[0168] In some optional embodiments, the acquisition module 41 is specifically used to extract relation triples based on the first data sample in the knowledge base, the relation triples being used to represent the relationship between the first entity and the second entity; acquire the second data sample based on the second entity; generate a multi-hop reasoning question based on the first data sample, the second data sample, and the relation triples, the multi-hop reasoning question being used to query based on the key information points corresponding to the second entity in the second data sample; and generate a checklist corresponding to each intermediate reasoning step based on the multiple intermediate reasoning steps corresponding to the multi-hop reasoning question and the standard answers associated with the intermediate reasoning steps.

[0169] In some optional embodiments, the acquisition module 41 is specifically used to extract multiple target triples based on the target entity of the third data sample in the knowledge base. The target triples are used to represent the relationship between the target entity and different tail entities. Based on the tail entities, several search keywords are acquired. Multiple fourth data samples are acquired through the search keywords. The fourth data samples include clue information of the tail entities. Based on the clue information of the tail entities and the reasoning relationship between the target entities, a multi-hop reasoning question is generated. The multi-hop reasoning question is used to identify the target entity based on the clue information of the tail entities. The target entity is the standard answer associated with the multi-hop reasoning question. Based on the target entity and the tail entities in each target triple, a checklist containing multiple check items is generated.

[0170] In some optional embodiments, each key information point corresponds to several check items, which are used to check whether the key information point exists and / or whether the key information point is accurate.

[0171] This specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method described in any of the above embodiments.

[0172] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the above embodiments.

[0173] This specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0174] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0175] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0177] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0178] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for evaluating large language models, the method comprising: Obtain an evaluation set, which includes user questions and the intent categories corresponding to the user questions; The user's question is input into the large language model, and the answer output by the large language model is obtained. Based on the intent category of the user's question, several metrics to be evaluated are determined; The answer is evaluated based on the various indicators to be evaluated, and the evaluation results are obtained.

2. The method according to claim 1, wherein, Based on the intent category of the user's question, several metrics to be evaluated are determined, including: Based on the intent category of the user's question, at least one high-level indicator to be evaluated is determined, including richness, usability, insight, and inspiration. The evaluation of the response based on each of the indicators to be evaluated, to obtain the evaluation result, includes: The response is evaluated based on at least one of the aforementioned advanced and basic metrics to be evaluated, and an evaluation result is obtained. The basic metrics include factuality, completeness, and relevance.

3. The method according to claim 1, wherein, The evaluation of the response based on each of the indicators to be evaluated, to obtain the evaluation result, includes: For each of the aforementioned indicators, the answer is scored using the evaluation method corresponding to the indicator to obtain a scoring result. The evaluation method includes at least one of the following: checklist-based evaluation, scoring rule-based evaluation, and fact-checking. The evaluation results are obtained based on the scoring results of each of the aforementioned indicators.

4. The method according to claim 1, wherein, The acquisition of the evaluation set includes: Obtain the user's question and the intent category of the user's question. An evaluation set is constructed based on the user questions and intent categories.

5. The method according to claim 4, wherein, The intent category for obtaining the user's question includes: The user question is classified into intents using multiple primary models to obtain the corresponding user intent categories. The user intent category that appears most frequently is determined as the intent category of the user question.

6. The method according to claim 4, wherein, The evaluation set is constructed based on the user question and the intent category, including: Based on the checklist generation method corresponding to the type of user question, and based on the user question and the standard answer associated with the user question, a checklist containing multiple check items is generated, wherein the check items correspond to key information points in the user question and / or the standard answer; Based on the user question, the intent category, and the checklist, an evaluation set is constructed. The checklist in the evaluation set is used to check the key information points in the answer.

7. The method according to claim 6, wherein, The user question is of any one of the following types: open-ended question, time-sensitive question, or multi-hop reasoning question; wherein, the open-ended question is a question without a fixed answer in a real-world scenario, the time-sensitive question is a question that changes over time, and the multi-hop reasoning question is a question that requires multiple intermediate reasoning steps to solve.

8. The method according to claim 7, wherein, The user acquisition issue includes: Obtain news data, and generate the timeliness question based on key information points in the news data. The key information points include entities, time, location, and data. The news data contains standard answers associated with the timeliness question.

9. The method according to claim 8, wherein, The method for generating a checklist based on the type of the user question, and based on the user question and the associated standard answer, generates a checklist containing multiple check items, including: The timeliness issue and the news data are input into the second major model, which generates a checklist containing multiple check items.

10. The method according to claim 7, wherein, The method for generating a checklist based on the type of user question, and based on the user question and the associated standard answer, generates a checklist containing multiple check items: Based on the aforementioned open-ended question, several third-party models generate multiple reference answers; Based on the multiple reference answers, a standard answer associated with the open-ended question is generated; Based on the key information points in the standard answer, check items corresponding to the key information points are generated to obtain a checklist.

11. The method according to claim 7, wherein, The user acquisition issue includes: Based on the first data sample in the knowledge base, relation triples are extracted, which are used to represent the relationship between the first entity and the second entity. Based on the second entity, obtain the second data sample; Based on the first data sample, the second data sample, and the relation triplet, a multi-hop reasoning problem is generated. The multi-hop reasoning problem is used to query based on the key information points corresponding to the second entity in the second data sample. The method for generating a checklist based on the type of the user question, and based on the user question and the associated standard answer, generates a checklist containing multiple check items, including: Based on the multiple intermediate reasoning steps corresponding to the multi-hop reasoning problem and the standard answers associated with the intermediate reasoning steps, a checklist is generated for each intermediate reasoning step.

12. The method according to claim 7, wherein, The user acquisition issue includes: Based on the target entity of the third data sample in the knowledge base, multiple target triples are extracted. The target triples are used to represent the relationship between the target entity and different tail entities. Based on the tail entity, several search keywords are obtained; Multiple fourth data samples are obtained through the search keywords, and the fourth data samples include clue information of the tail entity; Based on the clue information of the tail entity and the reasoning relationship between the target entity, a multi-hop reasoning question is generated. The multi-hop reasoning question is used to identify the target entity based on the clue information of the tail entity. The target entity is the standard answer associated with the multi-hop reasoning question. The method for generating a checklist based on the type of the user question, and based on the user question and the associated standard answer, generates a checklist containing multiple check items, including: Based on the target entity and the tail entity in each target triple, a checklist containing multiple check items is generated.

13. The method according to claim 6, wherein, Each of the key information points corresponds to several check items, which are used to check whether the key information point exists and / or whether the key information point is accurate.

14. An apparatus for evaluating a large language model, the apparatus comprising: The acquisition module is used to acquire an evaluation set, which includes user questions and the intent categories corresponding to the user questions; The reasoning module is used to input the user's question into the large language model and obtain the answer output by the large language model; The determination module is used to determine several metrics to be evaluated based on the intent category of the user's question. The evaluation module evaluates the answer based on each of the indicators to be evaluated, and obtains the evaluation results.

15. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-13.

Images