Model evaluation method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a model evaluation method and device, electronic equipment and storage medium, relates to the technical field of computers, in particular to the fields of artificial intelligence, intelligent search, large language model and the like, and can be used in application scenarios such as search enhanced large model quantitative evaluation. The scheme is: obtaining a plurality of candidate evaluation questions; for each candidate evaluation question, generating a first search query and a second search query; respectively executing the first search query and the second search query to generate search results; verifying the search results based on reference information, and labeling each candidate evaluation question as one of a plurality of preset information availability levels to obtain a plurality of evaluation questions; inputting each evaluation question into the model to be evaluated to obtain search behavior data and model answer results; determining the comprehensive calibration score of the model to be evaluated, and generating the search behavior calibration diagnosis result of the model to be evaluated. The present scheme can diagnose and evaluate the search enhanced model according to the actual availability level of the information.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, particularly to artificial intelligence, intelligent search, and large language models, and can be used in application scenarios such as search enhancement and large model quantitative evaluation. Specifically, it relates to model evaluation methods, devices, electronic devices, and storage media. Background Technology

[0002] Existing evaluation methods for large language models and search-enhanced large language models mainly include parametric knowledge-based fact-answering evaluation, deep web browsing-based search accuracy evaluation, synthetic retrieval environment-based search sub-capability decomposition evaluation, and standard retrieval set-based search engine quality evaluation. Parametric knowledge-based evaluation typically examines the model's knowledge coverage, answer accuracy, and confidence expression without invoking external tools through short factual questions, evaluating whether the model can express "don't know" when uncertain. Deep web browsing-based evaluation usually constructs challenging factual questions requiring multiple rounds of search, web browsing, and information integration to answer, with the model ultimately finding the correct answer as the primary evaluation criterion. Synthetic environment-based search sub-capability decomposition evaluation typically decomposes the search process into query generation, result selection, information extraction, and evidence integration in a controlled information environment, testing the model's performance in each stage. Search engine quality evaluation typically uses standard datasets to evaluate the relevance ranking of returned documents for a given query. Summary of the Invention

[0003] This disclosure provides a model evaluation method, apparatus, electronic device, and storage medium.

[0004] According to a first aspect of this disclosure, a model evaluation method is provided, comprising: acquiring multiple candidate evaluation items, each candidate evaluation item including question text and corresponding reference information; for each candidate evaluation item, extracting keywords from the question text, generating a first search query based on the keywords, and generating a second search query based on the reference information; executing the first search query and the second search query respectively to obtain a first search information set and a second search information set, and generating search results based on the first search information set and the second search information set; verifying the search results based on the information consistency between the reference information and the search results, generating verification results, and labeling each candidate evaluation item as one of a preset multiple information availability levels based on the verification results, thereby obtaining multiple evaluation items; inputting each evaluation item into the model being evaluated, so that the model being evaluated can answer each item, thereby obtaining search behavior data and model answer results; determining a comprehensive calibration score for the model being evaluated based on the search behavior data and model answer results corresponding to each information availability level, and generating a search behavior calibration diagnostic result for the model being evaluated based on the comprehensive calibration score.

[0005] According to a second aspect of this disclosure, a model evaluation apparatus is provided, comprising: a question acquisition module for acquiring multiple candidate evaluation questions, each candidate evaluation question including question text and corresponding reference information; a query construction module for extracting keywords from the question text for each candidate evaluation question, generating a first search query based on the keywords, and generating a second search query based on the reference information; a search execution module for executing the first search query and the second search query respectively, obtaining a first search information set and a second search information set, and generating search results based on the first search information set and the second search information set; and a verification and rating module for... The system ensures consistency between reference information and search results, verifies search results, generates verification results, and labels each candidate assessment item as one of several preset information availability levels based on the verification results, resulting in multiple assessment items. The model assessment module inputs each assessment item into the model being assessed, allowing the model to answer each item and generate search behavior data and model answer results. The result generation module determines the comprehensive calibration score of the model being assessed based on the search behavior data and model answer results corresponding to each information availability level, and generates search behavior calibration diagnostic results for the model based on the comprehensive calibration score.

[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods described in the embodiments of this disclosure.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0009] Using the scheme disclosed herein, a diagnostic evaluation of the search enhancement model can be performed according to the actual availability level of the information.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic flowchart of a model evaluation method according to an embodiment of the present disclosure; Figure 2 This is a schematic diagram of the structure of the evaluation execution system according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of an index calculation system according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of a question generation system according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of a model evaluation device according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of a scenario for a model evaluation method according to an embodiment of this disclosure; Figure 7 This is a structural diagram of an electronic device used to implement the model evaluation method of the embodiments of this disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] In this document, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document indicates any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this document refer to and distinguish between multiple similar technical terms, not to restrict the order or to limit there to only two. For example, "first feature" and "second feature" refer to two categories / two features; the first feature can be one or more, and the second feature can also be one or more.

[0014] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0015] Before introducing the technical solutions of the embodiments of this disclosure, the technical terms that may be used in this disclosure will be further explained: Search-enhanced large models refer to large language models with external search capabilities. After receiving a user's question, they can call search engines, web browsers, database retrieval tools, or other external information acquisition tools to find relevant information from the Internet or external knowledge sources, and then combine the search results to generate the final answer.

[0016] Information availability refers to the state in which an answer to a question or evidence supporting that answer exists, is easily found, is sufficiently consistent, and is sufficiently reliable on the public internet or in a specified search environment. In this embodiment of the disclosure, information availability does not simply characterize whether a search engine can return results, but rather refers to whether the model can obtain reliable evidence sufficient to support the answer from the accessible information environment.

[0017] Existing evaluation methods are insufficient to effectively measure whether the search behavior of search-enhanced large language models matches the real-world information environment. On one hand, some evaluation methods do not involve the use of search tools, making it difficult to reflect whether the model can reasonably decide whether to search, when to stop searching, and when to refuse answers when it has external search capabilities. On the other hand, some evaluation methods primarily focus on whether the model ultimately finds the correct answer, lacking coverage of scenarios such as unavailable information, contradictory information, and insufficient evidence. This makes it difficult to detect problems such as the model continuing ineffective searches when no answer exists on the public internet, fabricating answers when evidence is insufficient, and giving a single, definitive conclusion when authoritative sources contradict each other. Furthermore, while some methods can decompose and evaluate several sub-capabilities in the search process, their information environment is usually artificially constructed or controlled synthetic, differing from the dynamic, noisy, complex, and potentially conflicting information state of the real internet. Traditional retrieval evaluations primarily focus on the document ranking quality of the retrieval system and do not directly evaluate the degree of behavioral calibration of end-to-end search-enhanced large language models in terms of search decision-making, search frequency control, evidence synthesis, refusal behavior, and confidence expression.

[0018] In order to at least partially address one or more of the above-mentioned problems and other potential problems, this disclosure proposes a model evaluation method that can diagnostically evaluate search enhancement models according to the actual availability level of information.

[0019] This disclosure provides a model evaluation method. Figure 1This is a flowchart illustrating a model evaluation method according to an embodiment of the present disclosure. This model evaluation method can be applied to a model evaluation device. The model evaluation device is located in an electronic device. The electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. Mobile devices include, but are not limited to, search-enhanced large language model quantization evaluation devices, which can be mobile phones, tablets, etc. In some possible implementations, the model evaluation method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the model evaluation method includes: S101. Obtain multiple candidate assessment questions.

[0020] S102. For each candidate assessment question, extract keywords from the question text, generate a first search query based on the keywords, and generate a second search query based on the reference information.

[0021] S103. Execute the first search query and the second search query respectively to obtain the first search information set and the second search information set, and generate search results based on the first search information set and the second search information set.

[0022] S104. Based on the consistency of information between reference information and search results, verify the search results, generate verification results, and mark each candidate assessment item as one of the preset multiple information availability levels according to the verification results, thereby obtaining multiple assessment items.

[0023] S105. Input each assessment question into the model being assessed, so that the model can answer each assessment question and obtain search behavior data and model answer results.

[0024] S106. Based on the search behavior data and model response results corresponding to each information availability level, determine the comprehensive calibration score of the model being evaluated, and generate the search behavior calibration diagnosis result of the model being evaluated based on the comprehensive calibration score.

[0025] Here, candidate evaluation items include question text and corresponding reference information. Question text refers to the natural language question content used as input to the model being evaluated, i.e., the question portion of the evaluation item; reference information refers to the baseline information corresponding to the question text, which is not necessarily limited to a standard answer, but can also be a set of evidence used to determine whether an answer exists, whether it is consistent, or whether there is any controversy.

[0026] In this embodiment of the disclosure, multiple candidate assessment questions can be collected from existing question-and-answer data, existing question banks, public web pages, news reports, paper databases, government announcements, corporate announcements, or other trusted data sources. Furthermore, the candidate assessment questions can be formatted and standardized, for example, by removing duplicate questions, standardizing time formats, standardizing entity names, filtering unclear questions, and supplementing missing reference sources.

[0027] Here, keywords refer to words or phrases extracted from the question text that represent the core search intent of the question.

[0028] In this embodiment, natural language processing can be performed on the question text of each candidate assessment question to extract keywords such as core entities, time information, attribute words, event words, and limiting conditions. Furthermore, a first search query can be generated based on the keywords in the question text, primarily simulating the search expression that an ordinary user might input into a search engine based on the question itself. Simultaneously, a second search query can be generated based on reference information, primarily simulating the search expression that a professional user might input into a search engine based on their domain knowledge.

[0029] In this embodiment of the disclosure, searches can be performed based on a first search query and a second search query, respectively. Further, for the first search query, search results directly related to the question text are obtained to form a first search information set; for the second search query, search results related to reference information, reference answers, or reference sources are obtained to form a second search information set. Subsequently, the first and second search information sets can be cleaned and normalized, and the two sets can be aggregated to generate search results.

[0030] In this embodiment of the disclosure, the consistency between the reference information and the search results can be verified to determine whether the search results support the reference information, contradict the reference information, or fail to find sufficient relevant evidence. If the search results contain evidence from a reliable source that is consistent with the reference information, the candidate assessment item can be considered to have high information availability. If consistent evidence can only be found through a second search query, multiple query variations, or deeper web browsing, it can be marked as an information availability level requiring in-depth search. If the search results contain multiple authoritative sources that provide inconsistent answers, it can be marked as contradictory information. If no valid evidence supporting the reference information is found within a preset number of searches, a preset search depth, or a preset time range, it can be marked as information unavailable or low availability.

[0031] In this embodiment of the disclosure, each assessment question labeled with an information availability level can be input into the model being assessed, so that the model can generate a final answer based on the input question text. Specifically, all search behavior data can be recorded during model operation, including the number of search calls, query content, returned results, web pages read, search rounds, search time, and reasons for stopping the search; at the same time, the model's output answer results can be recorded, including the final answer, cited evidence, confidence level expression, whether the answer was rejected, whether it indicates insufficient information, and whether source conflicts were identified.

[0032] In this embodiment of the disclosure, the search behavior data and model answer results of the evaluated model can be grouped, statistically analyzed, and scored according to the information availability level of each assessment item. For example, indicators such as answer correctness score, search efficiency score, rejection reasonableness score, contradiction identification score, confidence calibration score, and excessive search penalty item can be calculated and weighted according to preset weights to obtain a comprehensive calibration score for the evaluated model. Finally, based on the comprehensive calibration score and the performance of each item under each information availability level, a search behavior calibration diagnostic result for the evaluated model can be generated, indicating, for example, whether the model has problems such as excessive searching, insufficient searching, excessively high confidence, insufficient rejection ability, insufficient identification of contradictory information, or insufficient evidence integration ability.

[0033] The technical solution of this disclosure, through a complete process from candidate question acquisition, dual-path search query generation, search result verification, information availability level labeling, model search behavior collection, to comprehensive calibration score generation, achieves a systematic evaluation of the behavioral performance of a search-enhanced large language model under different information environments. It can not only evaluate whether the model's final answer is correct, but also further evaluate whether the model searches reasonably, whether it stops invalid searches in a timely manner, whether it refuses to answer when information is unavailable, whether it expresses uncertainty when information is contradictory, and whether its confidence level matches the actual information availability. Simultaneously, it can more comprehensively diagnose the degree of calibration of the search behavior of the search-enhanced large language model, reduce the risk of excessive searching and fabricated answers, and provide quantitative basis for model optimization, system deployment, and reliability assessment.

[0034] In some embodiments, executing a first search query and a second search query respectively to obtain a first search information set and a second search information set, and generating search results based on the first search information set and the second search information set, includes: executing the first search query and the second search query respectively to obtain a first search information set and a second search information set corresponding to the first search query and the second search information set respectively; obtaining the first K search information items from the first search information set and the second search information set respectively to construct a target search set; wherein K is an integer not less than 1; extracting the title, link address, summary text and sorting position from each search information item in the target search set to generate search results.

[0035] In this embodiment of the disclosure, multiple types of search queries can be automatically constructed for each candidate assessment question to obtain search evidence for judging the availability of information. For example, firstly, core keywords are extracted from the question text of the candidate assessment question, and these core keywords are combined to generate a first type of search query, namely a naive query, used to simulate the behavior of an ordinary user searching solely based on the question description without knowing the answer. Subsequently, a second type of search query, namely an expert query, can be constructed based on the reference information or expected answer of the candidate assessment question. This expert query may include the expected answer, the name of the reference source, key evidence phrases, specific entity aliases, precise time, or other qualifiers that can improve the search hit rate, used to simulate the precise search strategy adopted by an expert user with certain background knowledge or answer clues.

[0036] In this embodiment of the disclosure, after generating the naive query and the expert query, the search engine's application programming interface (API) can be called to perform the retrieval and obtain the top-K search results for each query. Here, K can be set according to the evaluation accuracy and search cost, for example, from 5 to 10. For example, a smaller K value can be used when it is necessary to control the evaluation cost; a larger K value can be used when it is necessary to improve the recall rate and validation sufficiency.

[0037] In this embodiment of the disclosure, after constructing the target search set, preset fields can be extracted from each search message in the target search set, including at least the title, link address, summary text, and sort position. The title reflects the topic of the webpage or document corresponding to the search result; the URL identifies the network location of the search result and can be used to access the webpage content or confirm the source; the summary text is usually a webpage fragment returned by the search engine and can be used to initially determine whether the search result is related to the question text or reference information; the sort position indicates the ranking of the search information in the original search result list and can be used to measure whether the relevant information is easily recalled by the search engine. Furthermore, the query type, sort position, source site, and retrieval time corresponding to the search information can also be recorded. If the search information in the target search set comes from a first search information set or a second search information set, its corresponding query source identifier can also be retained to distinguish whether the search information was recalled by the first search query or the second search query. Finally, the extracted title, link address, summary text, and sort position can be organized according to a unified data structure to generate search results.

[0038] Thus, by constructing both naive and expert queries simultaneously, we can simulate two typical search paths: ordinary users searching based on questions and professional users conducting precise searches based on answer clues. This allows for a more comprehensive assessment of whether candidate assessment questions can be found in a public search environment. By selecting only the top K search results, we can control the scale of search result processing while ensuring coverage of the main relevant results, reducing computational costs and evaluation latency. Extracting structured fields such as titles, link addresses, abstract text, and sorting positions provides a standardized data foundation for subsequent information consistency verification, source tracing, relevance assessment, and availability level labeling, thereby improving the automation and stability of the assessment question construction process.

[0039] In some embodiments, after executing the first search query and the second search query respectively to obtain the first search information set and the second search information set, and generating search results based on the first search information set and the second search information set, the method further includes: responding to the fact that the reference information of the candidate assessment questions includes at least two contradictory reference claims, constructing an opposing viewpoint search query based on the contradictory reference claims; executing the opposing viewpoint search query to obtain an opposing viewpoint search information set; obtaining the top K search information from the opposing viewpoint search information set to construct an opposing viewpoint search set; wherein K is an integer not less than 1; extracting the title, link address, summary text and sorting position from each opposing viewpoint search information in the opposing viewpoint search set and adding it to the search results.

[0040] Here, contradictory reference claims refer to two or more reference statements in the reference information of a candidate assessment item that give different conclusions on the same issue or the same fact to be verified and cannot be valid at the same time. These statements can indicate that the candidate assessment item may be in an environment of conflicting or controversial information.

[0041] In this embodiment of the disclosure, it can be further determined whether the reference information of the candidate assessment question includes at least two contradictory reference claims. For example, answer values, factual statements, or conclusive statements from multiple sources can be extracted from the reference information, and the entities, attributes, times, values, and relationships involved can be compared. If multiple reference sources give different answer values ​​for the same question, or if natural language reasoning results indicate a contradictory relationship between the two claims, then it is determined that the reference information of the candidate assessment question includes at least two contradictory reference claims.

[0042] In this embodiment of the disclosure, after determining that the reference information of the candidate assessment question includes at least two contradictory reference claims, an opposing viewpoint search query can be constructed based on these at least two contradictory reference claims. For example, at least two mutually contradictory reference claims can be extracted from the reference information of the candidate assessment question, and corresponding search queries can be constructed around each contradictory reference claim. For instance, the disputed entity, disputed attributes, claim content, source name, and words such as "dispute," "different statements," and "conflict" can be combined to form an opposing viewpoint search query for retrieving different viewpoints or sources of contradiction.

[0043] In this embodiment, the search engine's API can be invoked to perform a retrieval and obtain the top-K search results for each query. K can be set according to the evaluation accuracy and search cost, for example, from 5 to 10. Subsequently, for each search result, its title, URL, and summary fragment can be collected, and the query type, sorting position, source site, and retrieval time corresponding to that search result can be further recorded. Then, the obtained top-K search results are aggregated, deduplicated, and formatted, and added to the search result set corresponding to the candidate evaluation question.

[0044] Thus, when there are at least two contradictory reference claims in the reference information of candidate assessment questions, a search query for opposing viewpoints is further constructed and executed. Search results related to different claims, disputed information and sources of conflict are added to the search results. This can proactively retrieve the sources of evidence corresponding to different contradictory claims and avoid missing disputed information or misjudging the availability of information due to a single query path.

[0045] In some embodiments, the search results are verified based on the consistency between the reference information and the search results to generate a verification result, including: extracting key reference semantics from the reference information and extracting key result semantics from the search results; matching the key reference semantics with the key result semantics, and determining the hit information based on the matching result; the hit information includes at least one of string hit, keyword fragment hit, numerical equivalence hit, and semantic approximation hit; generating a verification result based on the search results corresponding to the hit information; the verification result includes at least one of the search query term, search result ranking position, summary text, and link address.

[0046] Here, key reference semantics refers to semantic units extracted from the reference information of candidate assessment questions that can characterize the core meaning of the standard facts, reference answers, or reference evidence for that question. Key result semantics refers to the core semantic content extracted from search results that is related to the question text or reference information. Hit information refers to information records that show a matching or correspondence between key reference semantics and key result semantics, used to indicate the degree to which search results cover or support the reference information.

[0047] In this embodiment of the disclosure, answer matching detection can be performed on each search result returned by the search engine to determine whether the expected answer to the candidate assessment question can be directly obtained from the search result summary fragment. Specifically, after obtaining the search results corresponding to each search query, a summary fragment can be extracted from each search result and used as the key result semantics. Subsequently, the expected answer or the key reference semantics of the answer can be determined based on the reference information of the candidate assessment question, such as answer entities, numerical values, time, location, concluding phrases, or other content that can represent the core meaning of the answer, and it can be detected whether the key reference semantics appears in the corresponding key result semantics.

[0048] Specifically, the matching process can employ a combination of exact string matching and semantic approximation matching. Exact string matching determines whether the expected answer or key answer fragment appears directly in the summary fragment with the same character format. Semantic approximation matching determines whether there is content in the summary fragment that is different from the expected answer but semantically equivalent or similar. For example, for numerical answers, the number format, unit, capitalization, Chinese / English expression, and unit of measurement can be normalized to identify "5 grams" and "5g" as a match. For content such as time, names, or abbreviations, equivalent expressions can be identified through format conversion, alias dictionaries, or semantic similarity models. If a summary fragment meets the preset matching conditions with the expected answer or key answer fragment, the search result can be identified as a hit, and its ranking in the corresponding query result list can be recorded.

[0049] In this embodiment of the disclosure, after determining the hit information, a verification result can be generated based on the search results corresponding to the hit information. Specifically, for each hit search result, at least one of the following information can be recorded: the corresponding search query term, the search result ranking position, the summary text, and the link address. The search query term indicates whether the search result was retrieved by a general search query, an expert search query, or an opposing viewpoint search query; the search result ranking position indicates the visibility and accessibility of the evidence in the search results; the summary text provides a content fragment that directly verifies the reference information; and the link address is used to locate the source of the evidence and support subsequent verification. Finally, the above records can be integrated into a verification result.

[0050] Thus, by extracting key semantics from both reference information and search results, and performing consistency verification through multiple methods such as string matching, keyword fragment matching, numerical equivalence matching, and semantic approximation matching, we can avoid missed or false judgments caused by relying solely on simple literal matching, thereby improving the ability to recognize different expressions, numerical formats, and synonyms. Simultaneously, by recording the search query terms, ranking positions, summary texts, and link addresses encountered during the verification process, we can preserve the source, visibility, and verifiability of search evidence. This provides a structured basis for subsequent information availability grading, contradictory information identification, and quality control of assessment questions, thereby improving the accuracy, stability, and interpretability of the assessment data construction for the search-enhanced large language model.

[0051] In some embodiments, the verification of search results based on the consistency between reference information and search results, and the generation of verification results, further includes: extracting the claim texts corresponding to multiple information sources and the authority level of each information source from the search results; performing a consistency comparison on the claim texts corresponding to multiple information sources; in response to the existence of contradictory conclusions, different values, negation relationships, or mutually exclusive conditions in the claim texts corresponding to at least two information sources, determining contradictory information based on the search results; the contradictory information includes at least one of information source, claim text, link address, and contradiction type; and adding the contradictory information and authority level to the verification results.

[0052] Here, the authority level of an information source refers to the credibility or authority rating assigned to each information source based on the source attributes of the webpage or document corresponding to the search result, the credibility of the publishing entity, the standardization of content publication, historical reliability, and the relevance to the field of the candidate evaluation questions. This rating is used to indicate the reference value of the information carried by a search result in the subsequent verification process.

[0053] In this embodiment, search results can be parsed to identify the information source corresponding to each search result, and then an authority level can be determined for each information source. The authority level can be determined according to preset source classification rules, or it can be calculated by combining a source whitelist, domain name type, publishing entity type, content field, historical credibility, citation frequency, and manual annotation results.

[0054] In this embodiment of the disclosure, claims related to candidate questions can be identified from each summary fragment. Subsequently, the claims in the summary fragments of different search results can be compared to determine whether two or more search results contain semantically contradictory or inconsistent claims. For example, for usage advice questions, if one search result's summary fragment contains positive claims such as "recommended," "can be used," or "suitable for use," while another search result's summary fragment contains negative claims such as "not recommended," "prohibited," or "unsuitable for use," then it can be determined that there is a semantically contradictory signal between the two. Similarly, for numerical questions, if different search result summary fragments provide different values ​​for the same object and the same indicator—for example, value A and value B are inconsistent, and they cannot be interpreted as the same value through unit conversion or format normalization—then it can be determined that there is a contradictory signal of numerical inconsistency.

[0055] Furthermore, contradiction detection can be performed by combining keyword opposition pattern matching and numerical inconsistency detection. Keyword opposition pattern matching identifies opposing expressions in the abstract fragment, such as affirmation versus negation, recommendation versus disrecommendation, permission versus prohibition, existence versus non-existence, and validity versus invalidity. Numerical inconsistency detection extracts numerical values, units, time ranges, and indicator names from the abstract fragment and compares the consistency of corresponding values ​​in different results after standardizing units and formats. When at least two search results are found to contain semantically contradictory or numerically inconsistent claims in their abstract fragments, the candidate topic is recorded as a contradiction signal. The corresponding search result ranking, abstract fragment, link address, contradictory claim text, and contradiction type are included as contradiction information, and the authority level of the contradiction information and related information sources are added to the verification results.

[0056] Thus, by extracting the claim texts from multiple information sources from search results and comparing their consistency based on authority levels, it is possible to effectively identify contradictory conclusions, different values, negation relationships, or mutually exclusive conditions between different sources, thereby discovering information conflicts that are difficult to identify using ordinary answer matching methods. Introducing authority levels allows for the differentiation between substantive conflicts between authoritative sources and noisy conflicts caused by low-credibility sources, helping to more accurately identify contradictory candidate assessment questions and improving the reliability of subsequent information availability grading, question quality control, and model fact-checking capability assessment.

[0057] In some embodiments, the multiple information availability levels include: a first information availability level, a second information availability level, a third information availability level, and a fourth information availability level; each candidate assessment item is labeled as one of the preset multiple information availability levels according to the verification result, resulting in multiple assessment items, including: in response to the verification result indicating that there is contradictory information from at least two information sources in the search results, labeling the corresponding candidate assessment item as the third information availability level; in response to the verification result indicating that there is no contradictory information and that reference information is hit in the search results corresponding to the first search query, labeling the corresponding candidate assessment item as the first information availability level. Information availability is categorized into four levels: Level 1: Information availability; Level 2: Information availability; Level 3: Information availability; Level 4: Information availability; Level 5: Information availability. If the verification result indicates that there is no contradictory information, and the search results for the first and second search queries do not contain the reference information, the corresponding candidate assessment topic is marked as having a Level 4 information availability. Candidate assessment topics marked as having a Level 1, Level 2, Level 3, or Level 4 information availability are then used as assessment topics.

[0058] In this embodiment of the disclosure, an information availability level system can be pre-set, which includes a first information availability level (L1), a second information availability level (L2), a third information availability level (L3), and a fourth information availability level (L4).

[0059] L1 means information search is available immediately, that is, the answers to the candidate assessment questions are accurate and specific factual information, and can be obtained directly from the top of the search results with a single ordinary search. For example, the answer can be identified from the title, summary fragment or linked page in the first three search results corresponding to the first search query.

[0060] L2 indicates that the information requires deep searching. That is, the answer does exist on the public Internet, but it is difficult to find directly by ordinary search queries. More professional search terms, multi-step search strategies, specific source searches, or specific chapters of the original document are required to obtain it.

[0061] L3 indicates that there is conflicting information, meaning that a search can reveal at least two sources of information that give different or even contradictory claims on the same issue. When answering such questions, the model should be able to identify conflicts, compare the authority of the sources, and express uncertainty.

[0062] L4 indicates that the information is structurally unavailable, meaning that the information being queried is not only difficult to search, but also structurally non-existent on the public Internet due to institutional restrictions, confidentiality regulations, non-disclosure mechanisms, or the fact that the data itself has not been publicly released.

[0063] In this embodiment of the disclosure, the information availability level of candidate assessment questions can be determined sequentially in descending order of priority.

[0064] Specifically, since the existence of contradictory information is a qualitative feature of the question, even if one side's claim can be directly obtained from the top of the search results, the core evaluation value of the question still lies in examining whether the model can identify conflicting information, compare the authority of sources, and express uncertainty. Therefore, the determination of contradictory signals takes precedence over the determination of the answer's position. For example, the highest priority L3 determination, i.e., contradictory-first determination, can be performed first. Claim extraction and consistency comparison can be performed on multiple summary fragments in the search results set to determine whether there are two or more information sources giving semantically inconsistent claims on the same issue. If the above contradictory signal is detected, the candidate evaluation question is marked as L3 regardless of whether an answer match has already appeared in the search results.

[0065] Furthermore, if no contradictory signals are detected, an L1 criterion is applied. For example, the search results for a regular search query, especially the top few results for a naive query (e.g., the top-3 results), can be checked to determine if an answer match exists. Answer matching can include exact string matching, key answer fragment matching, numerical equivalence matching, or semantic approximation matching. If the expected answer or key answer fragment is directly matched in the top-3 results of the naive query, it indicates that the answer to the candidate assessment question can be obtained directly through a single regular search, and the system labels this candidate assessment question as L1, i.e., a search-and-get-it question.

[0066] Furthermore, if no contradictory signals are detected and the answer is not found in the top search results of the naive query, an L2 criterion is applied. For example, it can be checked whether the top-5 search results of the naive query do not contain the reference answer; if it is confirmed that the ordinary search results do not directly provide the answer, it can be further determined whether the search results corresponding to the expert query contain the expected answer, key fragments of the answer, or content semantically equivalent to the reference answer. If the top-5 results of the naive query do not contain the answer, but the expert query does, it indicates that the answer to the candidate assessment question does indeed exist on the public internet, but requires a more precise search strategy, multi-step retrieval, specific source limitation, or professional query expression to find. Therefore, this candidate assessment question can be labeled as L2, i.e., a question requiring deep search.

[0067] Furthermore, if no contradictory signals are detected, and all search results for all search queries fail to find the answer, then an L4 determination is made. For example, all query results, including naive and expert queries, can be examined. If none of these results contain the expected answer, key fragments of the answer, equivalent values, or semantically similar content, then the candidate assessment item is marked as an L4 candidate, i.e., a structured unavailable item.

[0068] Finally, after labeling the candidate assessment items, further verification can be conducted using manual review, domain rules, or publicly available source mechanisms to determine whether the information truly belongs to the corresponding information availability level. Then, the complete automatic verification process can be structured and stored as supplementary verification metadata for the items.

[0069] Thus, by categorizing candidate assessment questions into four information availability levels, we can more precisely differentiate the focus of different questions on the model's capabilities. By prioritizing and judging questions in one go, we can avoid misclassifying questions with conflicting sources as ordinary factual questions, and distinguish between questions that can be answered with simple searches and those requiring complex search strategies. Furthermore, when search engine rankings change, webpage content is updated, links become invalid, or information timeliness diminishes, the same or updated query strategy can be re-run, and the verification metadata and availability tags can be updated based on the new search results, thereby improving the stability, maintainability, and timeliness adaptability of the assessment question bank.

[0070] In some embodiments, each assessment question is input into the model being assessed, so that the model can answer each assessment question to obtain search behavior data and model answer results. This includes: constructing assessment prompts based on any assessment question and a preset prompt template; inputting the assessment prompts into the model being assessed; receiving a tool call request output by the model being assessed and calling a search tool according to the tool call request; inputting the search content returned by the search tool into the model being assessed and generating search behavior data according to the tool call request and the search content; and obtaining the model answer results based on the output results of the model being assessed.

[0071] Here, a tool request refers to a structured request sent by the model being evaluated to the evaluation system to invoke external tools while answering evaluation questions. This indicates that the model believes its own parameter knowledge is insufficient to complete the answer and requires external retrieval capabilities to obtain the latest or more accurate information. A search tool refers to an external information retrieval interface provided to the model being evaluated, used to retrieve relevant content from the public internet, search engines, specified databases, web page indexes, document libraries, or preset retrieval environments based on the model's query request. Search tools can be actual search engine interfaces, encapsulated retrieval services, or offline search indexes built for stability evaluation.

[0072] In this embodiment of the disclosure, assessment prompts consisting of a preset prompt word template and assessment questions can be constructed first. The preset prompt word template is used to generate behavioral constraints, such as requiring the model to search before answering, and to execute strategies such as directly answering, pointing out contradictions, or refusing to fabricate information under different circumstances, such as sufficient search results, contradictions, or unavailable information. At the same time, a confidence score of 1 to 10 is given in the final answer.

[0073] In this embodiment, the constructed assessment prompts can be input into the model being assessed. Upon receiving the prompts, the model can determine whether external information retrieval is necessary based on the question content and the prompt requirements. If the model determines that an answer can be given without searching, it can directly output the answer; otherwise, it can output a tool invocation request. This tool invocation request can be represented in a structured format, including fields such as the tool identifier to be invoked and search query terms.

[0074] In this embodiment, if the output includes a tool call request, the search tool interface is invoked to perform the search, the search results are returned to the model, and the next round of interaction continues. If the output does not include a tool call request, it is considered that the model has provided a final answer, and the evaluation of the question ends. In particular, to avoid the model from searching indefinitely, a maximum number of search rounds and a timeout for each question can be set. If the maximum number of rounds is reached without completing the answer, the process is forcibly terminated and the reason for termination is recorded.

[0075] In this embodiment, the search results can be fed back to the model being evaluated, enabling the model to read the search results, determine the reliability of the source, extract information related to the question, and decide whether to continue calling the search tool. Specifically, if the model continues to output new tool call requests, the steps of parsing the request, calling the search tool, and returning the search results can be repeated until the model no longer requests to call the tool and outputs the final answer.

[0076] In this embodiment of the disclosure, search behavior data can be generated based on tool call requests and search content. This search behavior data is used to characterize the retrieval behavior and tool usage process of the model being evaluated when answering evaluation questions. Furthermore, when the model being evaluated outputs a final answer, the answer content, explanations, sources of citation, expressions of uncertainty, content of refusal to answer, or evidence can be extracted and used as the model's answer result for that evaluation question.

[0077] Figure 2 A schematic diagram of the evaluation execution system in an embodiment of this disclosure is shown, as follows: Figure 2 As shown, a function call-based agent loop evaluation framework can be constructed to evaluate search augmentation models in a real search environment. Specifically, this evaluation framework includes a prompt word unit 201, a search tool interface 202, an agent loop controller 203, and a model under evaluation 204. The prompt word unit 201 is used to construct evaluation prompt words based on any evaluation question and a preset prompt word template. The search tool interface 202 provides the web_search tool to the model via function calls, and the model can output a tool call request containing the query terms. After receiving the request, the agent loop controller 203 calls the backend of the real search engine and feeds back the returned title, URL, and summary fragment to the model.

[0078] Thus, by allowing the evaluated model to proactively request tool calls based on prompts during the evaluation process, and to invoke search tools, record search content and search behavior data, the final answer result and the intermediate retrieval process of the model can be obtained simultaneously. By linking and recording tool call requests, search tool returned content, and final answer results, the evaluation process has stronger traceability and interpretability, which helps to conduct a more comprehensive and objective assessment of the model's search ability, evidence utilization ability, and factual answering ability under questions with different information availability levels.

[0079] In some embodiments, search behavior data includes at least the number of searches, and model response results include at least the response text and confidence level. Based on the search behavior data and model response results corresponding to each information availability level, a comprehensive calibration score for the evaluated model is determined, and a search behavior calibration diagnostic result for the evaluated model is generated based on the comprehensive calibration score. This includes: aggregating the search behavior data and model response results according to the information availability level to obtain search behavior data and model response results corresponding to each information availability level; determining the confidence score of the evaluated model for each information availability level based on the corresponding confidence level; determining the efficiency score of the evaluated model for each information availability level based on the corresponding number of searches; inputting the corresponding response text into a preset semantic judgment model for each information availability level to determine multiple response types, and determining the behavior score of the evaluated model based on the proportion of response types; and determining the comprehensive calibration score based on the confidence score, efficiency score, and behavior score.

[0080] In this embodiment of the disclosure, after the model call for each assessment question is completed, all assessment results can be grouped and aggregated according to the information availability level pre-labeled for each assessment question. For example, questions under L1, L2, L3, and L4 can be grouped into one group, and the search count, answer text, and confidence level of each group of questions can be counted to obtain search behavior data and model answer results corresponding to each information availability level.

[0081] In this embodiment, a search calibration curve can first be plotted based on the average confidence level under different information availability levels to determine whether the model confidence level decreases as information availability decreases. The monotonicity violation rate and the expected calibration error of the search version can also be calculated. Furthermore, based on the average number of searches or the number of searches per question at each level, it can be determined whether the model's search count falls within the corresponding reasonable search count range, thereby obtaining a search efficiency index to reflect whether the model has problems such as insufficient searching, excessive searching, or failure to terminate the search in a timely manner in scenarios where information is unavailable. Even further, a semantic judgment model can be used to identify the answer type of the answer text, and then the proportion of different answer types under each information availability level can be statistically analyzed to form an answer type distribution matrix. The actual answer type distribution is compared with the preset ideal answer type distribution to determine whether the model can provide a clear answer when information is readily available, indicate conflict when information is contradictory, and refuse to answer or state that it cannot be confirmed when information is unavailable.

[0082] After obtaining the confidence score, efficiency score, and behavioral score corresponding to each information availability level, the three scores can be weighted and fused according to preset weights to obtain the comprehensive calibration score for that level. Furthermore, the comprehensive calibration scores for each information availability level are summarized to obtain the overall comprehensive calibration score of the evaluated model. For example, the comprehensive calibration score can be expressed by the following formula: in, Indicates the overall calibration score. Indicates the confidence score. Indicates efficiency score. Indicates behavioral rating. , and These are the weighting coefficients. These weighting coefficients can be pre-configured according to specific application scenarios and can satisfy the following: .

[0083] Figure 3 A schematic diagram of the index calculation system in an embodiment of this disclosure is shown, as follows: Figure 3 As shown, a multi-dimensional calibration index calculation system can be set up to quantitatively evaluate the search behavior and response behavior of the search enhancement model under different information availability levels. Specifically, this subsystem may include a confidence extraction unit 301, a search frequency statistics unit 302, a behavior classification unit 303, and an index aggregation unit 304. The system includes the following components: a confidence extraction unit 301, which extracts the model's self-reported confidence from the model's response text, for example, by identifying confidence numbers or confidence levels in the response using regular expressions; a search count unit 302, which counts the number of times the model calls search tools when answering each assessment question; a behavior classification unit 303, which classifies the final response of the model into behaviors such as confident response, cautious response, contradiction identification, or refusal to answer; an indicator aggregation unit 304, which determines the confidence score of the model based on the corresponding confidence level for each information availability level; determines the efficiency score of the model based on the corresponding search count for each information availability level; inputs the corresponding response text into a preset semantic judgment model for each information availability level, so that the semantic judgment model can determine multiple response types and determine the behavior score of the model based on the proportion of response types; and determines a comprehensive calibration score based on the confidence score, efficiency score, and behavior score.

[0084] Thus, this approach not only evaluates the correctness of the model's final answer but also comprehensively diagnoses whether the model's search behavior is reasonable, whether the confidence level matches, and whether the answer strategy meets expectations by combining search frequency, confidence level, and answer behavior type under different information availability levels. By aggregating and scoring according to information availability levels, it is possible to identify different shortcomings of the model in scenarios with simple available information, deep search information, contradictory information, and structurally unavailable information, thereby generating more granular and interpretable search behavior calibration results.

[0085] In some embodiments, for each information availability level, the confidence score of the evaluated model is determined based on the corresponding confidence level, including: determining the average confidence level for all model responses corresponding to each information availability level; determining the confidence bias corresponding to each information availability level based on the average confidence level and the reference confidence level corresponding to that information availability level; determining the confidence monotonicity score based on the average confidence level corresponding to each information availability level; and determining the confidence score based on the confidence bias and the confidence monotonicity score.

[0086] Here, reference confidence level refers to the expected confidence level benchmark value pre-set for different information availability levels, used to measure whether the confidence level given by the evaluated model matches the availability of the question information.

[0087] In this embodiment, the model-reported confidence scores for all questions at each level can be statistically analyzed and averaged to obtain AvgConf(L1), AvgConf(L2), AvgConf(L3), and AvgConf(L4). Further, the reference confidence score RefConf for each level can be pre-configured according to specific application scenarios. For example, at a confidence score scale of 1 to 10, the default reference confidence score can be set to a decreasing sequence RefConf=[9,7,5,2], corresponding to L1, L2, L3, and L4, respectively. L1 corresponds to scenarios with high information certainty, hence a higher reference confidence score; L4 corresponds to scenarios where information is unavailable or should be rejected, hence a lower reference confidence score; the reference confidence scores for L2 and L3 can be linearly interpolated based on the degree of information certainty, or they can be calibrated by domain experts in conjunction with business tolerance.

[0088] In this embodiment of the disclosure, the confidence bias can be represented by a search-based ECE, which measures the deviation between the model's average confidence level and a preset reference confidence level at each information availability level. For example, the search-based ECE can be represented by the following formula: in, The number of information availability levels, for L1, L2, L3, and L4 in this embodiment of the disclosure, ; The model represents the first time. Average self-reported confidence level under each information availability level; This indicates the reference confidence level corresponding to that level. Specifically, The smaller the value, the closer the model's self-reported confidence level is to the expected reference confidence level, and the better the model's confidence level calibration effect.

[0089] Furthermore, the confidence monotonicity score can be determined based on the average confidence score corresponding to each information availability level. Confidence monotonicity measures whether the model's confidence reasonably decreases as information availability decreases. Generally, information for L1 questions is most readily available, and the model's average confidence score should be high; L2 questions require deep searching, and the average confidence score can be slightly lower; L3 questions contain contradictory information, and the model should reduce its certainty; L4 questions have unavailable public information, and the model's confidence score should be the lowest or close to the lowest. Scores can be calculated based on the number of adjacent levels satisfying the monotonic relationship, or deductions can be calculated based on the magnitude of violations of the monotonic relationship.

[0090] Finally, the confidence score of the model being evaluated can be determined based on the confidence bias and the confidence monotonicity score. For example, the confidence score can be expressed by the following formula: in, This indicates the confidence level monotonicity score.

[0091] Thus, by comparing the average confidence level with the reference confidence level, it is possible to determine whether the model is overconfident or underconfident on a certain type of question. Simultaneously, by using the monotonicity score of the confidence level, it is possible to determine whether the model can correspondingly decrease its confidence as information availability decreases. This not only identifies problems where the model answers incorrectly but also reveals calibration defects where the model still outputs high confidence in high-risk scenarios, thereby improving the interpretability and risk identification capabilities of the search augmentation model evaluation results.

[0092] In some embodiments, determining a confidence monotonicity score based on the average confidence level corresponding to each information availability level includes: arranging multiple information availability levels in descending order of information availability, comparing the average confidence levels corresponding to adjacent information availability levels; recording a confidence monotonicity anomaly in response to a lower information availability level's average confidence level not being lower than a higher information availability level's average confidence level; and determining a confidence monotonicity score based on the number of confidence monotonicity anomalies.

[0093] In some implementations, a calibration curve can be plotted using the four levels of information availability as the horizontal axis and the model's average self-reported confidence at each level as the vertical axis. The four levels of information availability can be represented as L1, L2, L3, and L4, with information availability decreasing progressively from L1 to L4. Ideally, the model's confidence should decrease as information availability decreases; that is, the calibration curve should show a decreasing trend of L1>L2>L3>L4, indicating that the model can reasonably adjust the certainty of its responses based on the availability and reliability of external information.

[0094] Furthermore, we can first compare adjacent information availability level pairs sequentially, including (L1, L2), (L2, L3), and (L3, L4). For any adjacent level pair, if the average confidence score of the higher information availability level is less than or equal to the average confidence score of the lower information availability level, it indicates that the model has failed to reflect the strict decrease in confidence score as information availability decreases, and a monotonicity violation is recorded. The confidence monotonicity score can be expressed as: in, This represents the number of adjacent information availability level pairs. For L1, L2, L3, and L4 in this embodiment of the disclosure, ; This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. This represents the confidence monotonicity score, ranging from 0 to 1. A lower value indicates better confidence monotonicity of the model. Specifically, when... When the value is 0, it means that the average confidence level of the model on L1 to L4 completely satisfies the strict decreasing relationship.

[0095] In this way, it is possible to determine whether the evaluated model can reasonably reduce its confidence level as information availability decreases, thus reflecting the model's ability to perceive the difficulty of the question information and the sufficiency of the evidence. By comparing the average confidence levels of adjacent levels and statistically analyzing monotonicity anomalies, the system can quickly identify problems where the model maintains excessively high confidence levels on difficult, contradictory, or unavailable questions. This helps to identify defects such as overconfidence, risky answers, and insufficient confidence calibration, making the model evaluation results more stable, intuitive, and interpretable.

[0096] In some embodiments, for each information availability level, the efficiency score of the evaluated model is determined based on the corresponding search count, including: determining a reasonable search count range based on the information availability level of each evaluation item; determining whether the search count of each evaluation item falls within the corresponding reasonable search count range; in response to the search count falling within the corresponding reasonable search count range, assigning a first search efficiency evaluation result to the corresponding evaluation item; in response to the search count not falling within the corresponding reasonable search count range, assigning a second search efficiency evaluation result to the corresponding evaluation item based on the degree of deviation of the search count from the reasonable search count range; and determining an efficiency score based on the first or second search efficiency evaluation result for each evaluation item.

[0097] Here, the reasonable range of search attempts refers to the pre-defined range within which the model should invoke search tools for assessment questions of different information availability levels. This range is used to determine whether the model's search behavior is appropriate, avoiding both over-searching on simple questions and under-searching on complex, contradictory, or unavailable questions.

[0098] In this embodiment of the disclosure, the efficiency score of the model being evaluated can be determined based on the number of searches for the corresponding assessment question for each information availability level. Specifically, the system first reads the information availability level of each assessment question and determines the reasonable search frequency range corresponding to that level according to a preset configuration. For example, for level L1, since question information is usually easy to obtain through searching, the model should be able to quickly locate the answer, so the reasonable search frequency can be set to 1 to 2 times; for level L2, since the question requires a certain depth of search, the model usually needs to obtain sufficient information through multiple keywords or multiple rounds of retrieval, so the reasonable search frequency can be set to 2 to 4 times; for level L3, since the question information may have different sources or conflicting content, the model needs to retrieve multiple sources and perform cross-validation, so the reasonable search frequency can also be set to 3 to 5 times; for level L4, since the question information itself is unavailable or difficult to verify through public channels, the model should promptly identify that the information is unavailable and stop searching after the necessary search, rather than continuing to search to the maximum number of rounds allowed by the system, so the reasonable search frequency can be set to 1 to 5 times.

[0099] In this embodiment of the disclosure, the number of searches performed by the model for each assessment question can be counted to evaluate whether the model's search efficiency matches the corresponding information availability level. Specifically, it can be determined whether the number of searches for each assessment question falls within a reasonable range that matches the information availability level corresponding to that assessment question. If the actual number of searches is within this range, it indicates that the model's search behavior matches the expected search depth for that type of question, and a first search efficiency evaluation result can be assigned to that assessment question. For example, the first search efficiency evaluation result can be expressed as... .

[0100] In this embodiment, if the actual number of searches does not fall within a reasonable search range, a second search efficiency evaluation result can be assigned to the assessment question based on the degree of deviation of the search count from the reasonable search range. For example, if the actual number of searches is lower than the lower limit of the reasonable search range, it indicates that the model may have an insufficient search problem, such as providing an answer after only one search in questions requiring multi-source verification; if the actual number of searches is higher than the upper limit of the reasonable search range, it indicates that the model may have an over-search problem, such as repeatedly searching in questions with readily available information. Furthermore, points can be deducted based on the distance from the lower or upper limit; the greater the deviation, the lower the second search efficiency evaluation result. For example, the reasonable search range for an L2 question is 2 to 4 times; if the model only searches once, it can be considered a slight search deviation. If there are 0 searches, and the deviation from the minimum number of searches is 2, it can be considered a moderate search deviation. If a search is performed 7 times, and the deviation from the highest search count is 3, then it can be considered a severe search deviation. .

[0101] In this embodiment of the disclosure, after obtaining the first or second search efficiency evaluation results for each evaluation item, the items under the same information availability level are summarized to determine the efficiency score corresponding to that level. Specifically, the efficiency evaluation results of each item can be converted into numerical scores, and the average value can be calculated to obtain the search efficiency score for that level; alternatively, the proportion of items with search counts falling within a reasonable range, the proportion of insufficient searches, and the proportion of excessive searches can be statistically analyzed, and the efficiency score can be determined based on the above proportions.

[0102] In this way, the number of model searches can be graded and evaluated based on the availability of question information, making the search efficiency assessment more consistent with actual search and question answering scenarios. By configuring reasonable search frequency ranges for different levels, the system can not only identify excessive searches by the model on simple questions, reducing invalid calls and time costs, but also identify insufficient searches by the model on complex, contradictory, or unavailable questions, reducing the risk of hasty and fabricated answers. Therefore, this method can measure whether the evaluated model has reasonable retrieval depth control capabilities from the perspective of search behavior, and provide interpretable diagnostic basis for subsequent model optimization.

[0103] In some embodiments, for each information availability level, the corresponding answer text is input into a preset semantic judgment model to determine multiple answer types, and the behavior score of the evaluated model is determined based on the proportion of answer types. This includes: inputting the answer text into the preset semantic judgment model to perform keyword matching and sentence structure matching on the answer text to obtain multiple answer types; determining the proportion of answer types corresponding to the information availability level based on each answer type; determining the expected proportion of types based on the information availability level; matching the proportion of answer types with the expected proportion of types, and determining the behavior score based on the matching result.

[0104] Here, "response type" refers to the behavioral category that the pre-defined semantic judgment model classifies for each response based on the semantic features, keyword features, and sentence structure features of the model's response text. This category describes the expression strategy adopted by the model being evaluated when answering questions. "Expected type ratio" refers to the expected distribution of the proportion of each type of response for each information availability level, pre-defined. This represents the distribution of response behaviors that the ideal model should present at a given information availability level.

[0105] In this embodiment of the disclosure, the final answers generated by the model under test for the test questions can be classified by answer type, and the model's answer behavior at different information availability levels can be evaluated based on the classification results to determine whether it meets expectations. For example, answer types can be classified into four types: confident answer, cautious answer, contradiction identification, and refusal to answer. A confident answer can be represented as `confident_answer`, which means the model under test directly gives a clear answer without expressing substantial uncertainty in the answer; a cautious answer can be represented as `hedged_answer`, which means the model under test uses expressions such as "maybe" or "not quite sure" to indicate that the answer has some uncertainty; contradiction identification can be represented as `acknowledge_contradiction`, which means the model under test points out contradictions, inconsistencies, or irreconcilable differences between different sources, different data, or different evidence in the answer; and refusal to answer can be represented as `refuse`, which means the model under test explicitly states that it could not find reliable information, the existing information is insufficient to support the answer, or it cannot give a definite answer based on reliable evidence.

[0106] In this embodiment of the disclosure, the complete answer text of the model being evaluated can be input into a preset semantic judgment model, which then classifies the answer based on preset prompts. For example, the prompts may include: "Please analyze the following search enhancement model's answer and determine which behavioral category it belongs to: A. Confident answer: Directly provides a clear answer without expressing substantial uncertainty; B. Cautious answer: Provides an answer but explicitly expresses uncertainty; C. Identifies contradictions: Points out contradictory or inconsistent information from different sources; D. Rejects answer: Explicitly indicates that reliable information could not be found. The judgment model needs to output a category letter and a one-sentence reason for the judgment." In this embodiment of the disclosure, after obtaining the final behavioral classification results for each assessment item, the classification results can be statistically analyzed according to the information availability level, and the distribution of the four behavioral types under each level can be calculated. For example, for level L1, since this level typically indicates readily available information and sufficient evidence, the ideal behavioral distribution should be dominated by confident answers, for example, the proportion of confident_answer should not be less than 80%; for level L2, since this level typically indicates that a certain depth of search is required but relatively reliable information can still be obtained, the ideal behavioral distribution can be dominated by confident and cautious answers, i.e., confident_answer and heeded_answer together account for a high proportion; for level L3, since this level typically indicates that there may be contradictions or inconsistencies between different sources, the ideal behavioral distribution should be dominated by identifying contradictions, for example, the proportion of acknowledge_contradiction should not be less than 60%; for level L4, since this level typically indicates that reliable information is unavailable or insufficient to support the answer, the ideal behavioral distribution should be dominated by refusal to answer, for example, the proportion of refuse should not be less than 60%. Furthermore, the actual behavioral distribution of the evaluated model at each information availability level can be compared with the ideal behavioral distribution described above to determine whether the model can adjust its response strategy based on information availability and reliability. If the model can primarily provide explicit answers at level L1, proactively point out information contradictions at level L3, and promptly refuse to answer or explain insufficient information at level L4, it indicates that the model has a good ability to calibrate search-enhanced response behavior. Conversely, if the model still outputs a large number of confident answers at levels L3 or L4, it suggests that the model may be at risk of ignoring source conflicts, being overconfident, or fabricating answers.

[0107] Thus, this approach not only focuses on the correctness of the model's answers but also assesses whether the model adopted an answering strategy commensurate with information availability from the semantic behavior level of the answer text. By pre-setting a semantic judgment model to identify answer types such as definitive answers, cautious answers, contradictory prompts, insufficient information, or refusal to answer, and comparing the proportion of actual answer types with the proportion of expected types, the system can effectively identify issues such as whether the model is overly conservative in information-rich scenarios, whether it can prompt conflicts in information-contradictory scenarios, and whether it can refuse to fabricate answers in information-unavailable scenarios. Therefore, this method can improve the granularity and interpretability of model evaluation, help identify behavioral biases of the evaluated model under different information availability conditions, and provide a reliable basis for subsequent model optimization, security control, and application deployment.

[0108] In some implementations... Figure 4 A schematic diagram of the structure of the question generation system in an embodiment of this disclosure is shown, such as... Figure 4As shown, a subsystem for constructing and automatically validating the evaluation question bank can be set up to construct a calibration evaluation question bank for the search enhancement model according to the availability of answer information on the public Internet. Specifically, this subsystem includes an L1 constructor 401, an L2 constructor 402, an L3 constructor 403, an L4 constructor 404, and a search validation unit 405, with each constructor generating candidate evaluation questions with different information availability levels.

[0109] For example, for L1 builder 401, questions that can be obtained through information search can be constructed. These questions can select specific factual information published within the last 6 to 18 months, such as institutional announcements, statistical data, annual reports, or recent announcements from regulatory agencies, to ensure that the information is sufficiently recent and specific, reducing the likelihood that the model being evaluated will directly rely on training data to answer. Searches can be performed using question keywords, and it can be confirmed that the standard answer appears in the summary fragments of the first three search results. The expected model behavior for L1 questions is to quickly obtain the answer within 1 to 2 searches and provide a correct answer with high confidence, for example, a confidence score in the range of 8 to 10.

[0110] Furthermore, for L2 constructor 402, questions requiring deep information search can be constructed. These questions can be generated using a reverse construction method: first, a specific and verifiable answer is determined from official documents, original regulations, or professional reports; then, it is verified that this answer will not appear in the top five search results of the most intuitive search; finally, questions requiring multiple hops or in-depth research are designed around this answer. The expected model behavior for L2 questions is to perform 2 to 4 deeper searches, provide the correct answer, and moderately reduce the confidence level compared to L1, for example, to a confidence level between 6 and 8.

[0111] Furthermore, for L3 constructor 403, questions with contradictory information types can be constructed. These questions originate from known disagreements among authoritative sources within a professional field, and the types of contradictions can include conflicts between old and new sources, jurisdictional differences, conflicts between principles and practices, regulatory and judicial disagreements, and academic controversies. Each candidate question can be searched and validated to confirm the existence of contradictory information from different sources in the search results, and the specific claims and authority levels of each source are recorded. For L3 questions, the evaluation criterion is not a single correct answer, but rather a contradictory description including the claims of source A, the claims of source B, the authority levels of each source, and the expected correct behavior. The expected model behavior of L3 questions is to identify the existence of contradictions, cite specific sources, explain the reasons for the differences between different sources, judge the authority of the sources, and express appropriate uncertainty, with the confidence level significantly reduced, for example, in the range of 5 to 7 points.

[0112] Furthermore, for the L4 constructor 404, questions of the type "structured unavailability of information" can be constructed. The information requested by this type of question should be reasonable and of genuine interest to professionals, rather than meaningless. Simultaneously, the reason for the unavailability of this information should be structural, determined by institutional, authorization, confidentiality rules, or release schedule, rather than being a matter of chance. L4 questions can include internal review information, unpublished data, confidential business information, information from ongoing confidentiality regulatory or judicial processes, and restricted data requiring payment, authorization, or qualifications to access. A search can be performed to confirm the existence of relevant information in publicly available results, demonstrating the reasonableness of the question and inducing the model to attempt a search, but the search results do not contain the specific information requested. Simultaneously, the system logically verifies that its unavailability is determined by institutional or structural reasons. The expected model behavior for L4 questions is to stop promptly after the necessary search, rather than continuing the search to the maximum round, and clearly state that no reliable publicly available information was found, providing a reasonable reason for refusal with extremely low confidence (e.g., in the range of 1 to 3), or to explicitly refuse to answer.

[0113] In addition, the search verification unit 405 can first generate a first search query and a second search query for each candidate assessment item, then execute the first search query and the second search query respectively to obtain the search results, and then verify the search results based on the reference information and the information consistency between the search results to generate a verification result, thereby obtaining the assessment item set.

[0114] In this way, evaluation questions can be constructed according to four scenarios: readily available information, requiring deep search, contradictory information, and structurally unavailable information. The availability tags of the questions are automatically verified using a search validation unit, ensuring the question bank has verifiability, hierarchy, and behavioral differentiation. The resulting evaluation question bank can not only test whether the search augmentation model can find the answers, but also further evaluate whether it can adjust search depth, confidence, and answering strategies according to different information availability scenarios.

[0115] In some implementations, after calculating the multidimensional calibration indicators, abnormal behaviors of the evaluated model under different information availability levels can be further identified, and corresponding diagnostic conclusions and optimization suggestions can be generated. Specifically, multiple abnormal search behavior patterns can be predefined, and corresponding detection rules, diagnostic conclusions, and optimization suggestions can be configured for each abnormality.

[0116] For example, when the model's average number of searches for L4 information-structured unavailable questions approaches the maximum number of search rounds, it can be identified as having an "L4 over-search" anomaly, indicating that the model's search termination ability is insufficient and it cannot promptly identify unavailable information. It is recommended to optimize the search termination strategy and add an information unavailability judgment mechanism. When the rejection rate in L4 questions is too low, it can be identified as having an "L4 rejection failure" anomaly, indicating that the model tends to generate answers even when reliable public information is lacking. It is recommended to strengthen the training signal of "rejecting answers when uncertain." When the contradiction identification rate in L3 information-contradictory questions is too low, it can be identified as having an "L3 contradiction neglect" anomaly, indicating that the model fails to effectively detect conflicts between different search results. It is recommended to increase training data for contradictory information scenarios and optimize the prompting strategy. When the average number of searches for L1 information-easily available questions is too high, it can be identified as having an "L1 inefficiency" anomaly, indicating that the model's search strategy for easily available information is redundant. It is recommended to optimize query construction capabilities and reduce unnecessary repeated searches.

[0117] Furthermore, confidence-related anomalies can be detected. For example, when the average confidence level corresponding to more readily available information is lower than that of less readily available information, a "confidence inversion" anomaly can be identified, indicating that the model's confidence calibration logic is unreasonable, and it is recommended to calibrate the correspondence between its uncertainty expression and the information environment. When the average confidence level is high across all information availability levels, a "global overconfidence" anomaly can be identified, indicating that the model has failed to adjust its confidence expression according to changes in the information environment, and it is recommended to lower the confidence expression threshold and enhance the uncertainty perception capability.

[0118] Specifically, during the diagnostic process, the indicators can first be aggregated according to the model and information availability level to calculate the average confidence level, average number of searches, and distribution ratio of each behavior category; then the aggregation results are compared with the preset anomaly detection rules to identify the triggered anomaly patterns; next, the severity of the anomaly is assessed based on the degree to which the indicators deviate from the threshold; finally, the anomaly patterns, severity, diagnostic conclusions, and optimization suggestions are summarized and sorted according to severity to generate a structured diagnostic report.

[0119] In some implementations, when evaluating multiple models simultaneously, a cross-sectional comparative analysis can be performed to compare the differences in indicators among different models at the same information availability level, identifying the relative strengths and weaknesses of each model. For example, if a model's contradiction identification rate at the L3 level is significantly lower than that of other models, it can be marked as "L3 relative weakness".

[0120] It should be understood that Figures 2 to 4 The schematic diagrams shown are merely illustrative and not limiting, and are scalable; those skilled in the art can use them as a basis. Figures 2 to 4Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0121] This disclosure provides a model evaluation device, such as... Figure 5 As shown, the device may include: a question acquisition module 501, used to acquire multiple candidate assessment questions, each candidate assessment question including question text and corresponding reference information; a query construction module 502, used to extract keywords from the question text for each candidate assessment question, generate a first search query based on the keywords, and generate a second search query based on the reference information; a search execution module 503, used to execute the first search query and the second search query respectively, obtain a first search information set and a second search information set, and generate search results based on the first search information set and the second search information set; and a verification and rating module 504, used to verify and rate based on the reference information. The system verifies the information consistency between search results, generates verification results, and labels each candidate assessment question as one of several preset information availability levels based on the verification results, thus obtaining multiple assessment questions. The model assessment module 505 is used to input each assessment question into the model being assessed, so that the model can answer each assessment question, thereby obtaining search behavior data and model answer results. The result generation module 506 is used to determine the comprehensive calibration score of the model being assessed based on the search behavior data and model answer results corresponding to each information availability level, and to generate the search behavior calibration diagnosis result of the model being assessed based on the comprehensive calibration score.

[0122] In some embodiments, the search execution module 503 includes: an information search submodule, configured to execute a first search query and a second search query respectively, to obtain a first search information set and a second search information set corresponding to the first search query and the second search query respectively; an information integration submodule, configured to obtain the first K search information items from the first search information set and the second search information set respectively, and construct a target search set; wherein K is an integer not less than 1; and a result construction submodule, configured to extract the title, link address, summary text and sorting position from each search information item in the target search set, and generate search results.

[0123] In some embodiments, the model evaluation device further includes: a contradiction query module 507 ( Figure 5 (Not shown in the image), the reference information used in response to candidate assessment questions includes at least two contradictory reference claims, and a search query for opposing viewpoints is constructed based on the contradictory reference claims; Contradictory search module 508 ( Figure 5 (Not shown in the image), used to perform opposing viewpoint search queries and obtain a set of opposing viewpoint search information; Contradictory integration module 509 ( Figure 5(Not shown in the image), used to obtain the top K search results from the opposing viewpoint search information set and construct the opposing viewpoint search set; where K is an integer not less than 1; Result addition module 510 ( Figure 5 (Not shown in the image) is used to extract the title, link address, summary text, and sorting position from each opposing viewpoint search information in the opposing viewpoint search set and add it to the search results.

[0124] In some embodiments, the verification rating module 504 includes: a semantic extraction submodule, used to extract key reference semantics from reference information and key result semantics from search results; a semantic matching submodule, used to match key reference semantics with key result semantics and determine hit information based on the matching results; the hit information includes at least one of string hit, keyword fragment hit, numerical equivalence hit, and semantic approximation hit; and a verification result submodule, used to generate verification results based on the search results corresponding to the hit information; the verification results include at least one of search query terms, search result ranking position, summary text, and link address.

[0125] In some embodiments, the verification rating module 504 further includes: a rating extraction submodule, used to extract the claim texts corresponding to multiple information sources and the authority rating of each information source from the search results; a text comparison submodule, used to perform consistency comparison on the claim texts corresponding to multiple information sources; a contradiction determination submodule, used to determine contradiction information based on the search results in response to the existence of opposite conclusions, different values, negation relationships, or mutually exclusive conditions in the claim texts corresponding to at least two information sources; the contradiction information includes at least one of the information source, claim text, link address, and contradiction type; and a result addition submodule, used to add the contradiction information and authority rating to the verification result.

[0126] In some embodiments, the multiple information availability levels include: a first information availability level, a second information availability level, a third information availability level, and a fourth information availability level; the verification rating module 504 includes: a third-level submodule, configured to, in response to a verification result indicating that contradictory information from at least two information sources exists in the search results, mark the corresponding candidate assessment item as a third information availability level; a first-level submodule, configured to, in response to a verification result indicating that there is no contradictory information and that reference information is hit in the search results corresponding to the first search query, mark the corresponding candidate assessment item as a first information availability level; a second-level submodule, configured to, in response to a verification result indicating that contradictory information does not exist and that reference information is hit in the search results corresponding to the first search query, mark the corresponding candidate assessment item as a first information availability level; and a second-level submodule, configured to, in response to a verification result indicating that contradictory information exists in the search results corresponding to the first search query, mark the corresponding candidate assessment item as a first information availability level; and a third-level submodule, configured to, in response to a verification result indicating that contradictory information exists in the search results corresponding to the first search query, mark the corresponding candidate assessment item as a first information availability level; and a fourth-level submodule, configured to, in response to a verification result indicating that contradictory information exists in the search results corresponding to the first search query, mark the corresponding candidate assessment item as a first information availability level; and a fifth-level submodule, configured to, in response to a verification result indicating that contradictory information exists in the search results corresponding to the first search query, mark the corresponding candidate assessment item as a first information availability level; and a sixth-level submodule, configured to, in response to a verification result indicating that contradictory information exists in the search results corresponding to the first search query, mark the corresponding candidate assessment item as a first information availability level; and a seventh-level submodule, configured to, in response to a verification result indicating that contradictory information exists in the search results corresponding to the first search query, mark the corresponding candidate assessment item as a first information availability level; and a fifth-level submodule, configured to, in response to a verification result indicating that contradictory information exists in the search results corresponding to The first submodule indicates that there is no contradictory information, the search results for the first search query do not match the reference information, and the search results for the second search query match the reference information. The second submodule then marks the corresponding candidate assessment questions as having a second information availability level. The third submodule, in response to a verification result indicating that there is no contradictory information and the search results for both the first and second search queries do not match the reference information, marks the corresponding candidate assessment questions as having a fourth information availability level. The fourth submodule, in response to a verification result indicating that there is no contradictory information and the search results for both the first and second search queries do not match the reference information, marks the corresponding candidate assessment questions as having a first, second, third, or fourth information availability level.

[0127] In some embodiments, the model evaluation module 505 includes: a prompt word construction submodule, used to construct evaluation prompt words based on any evaluation question and a preset prompt word template; a prompt word input submodule, used to input the evaluation prompt words into the model being evaluated; a tool invocation submodule, used to receive a tool invocation request output by the model being evaluated, and invoke a search tool according to the tool invocation request; a data processing submodule, used to input the search content returned by the search tool into the model being evaluated, and generate search behavior data according to the tool invocation request and the search content; and a result collection submodule, used to obtain the model's answer results based on the output results of the model being evaluated.

[0128] In some embodiments, the search behavior data includes at least the number of searches, and the model response results include at least the response text and confidence level. The result generation module 506 includes: a level division submodule, used to aggregate the search behavior data and model response results according to the information availability level to obtain search behavior data and model response results corresponding to each information availability level; a first scoring submodule, used to determine the confidence score of the evaluated model based on the corresponding confidence level for each information availability level; a second scoring submodule, used to determine the efficiency score of the evaluated model based on the corresponding number of searches for each information availability level; a third scoring submodule, used to input the corresponding response text into a preset semantic judgment model for each information availability level, so that the semantic judgment model can determine multiple response types, and determine the behavior score of the evaluated model based on the proportion of response types; and a scoring integration submodule, used to determine a comprehensive calibration score based on the confidence score, efficiency score, and behavior score.

[0129] In some embodiments, the first scoring submodule is configured to: determine the average confidence level for all model response results corresponding to each information availability level; determine the confidence deviation corresponding to each information availability level based on the average confidence level and the reference confidence level corresponding to that information availability level; determine the confidence monotonicity score based on the average confidence level corresponding to each information availability level; and determine the confidence score based on the confidence deviation and the confidence monotonicity score.

[0130] In some embodiments, the first scoring submodule is configured to: arrange multiple information availability levels in descending order of information availability, compare the average confidence levels corresponding to adjacent information availability levels; record a confidence monotonicity anomaly in response to the lower information availability level's average confidence level being no lower than the higher information availability level's average confidence level; and determine a confidence monotonicity score based on the number of confidence monotonicity anomalies.

[0131] In some embodiments, the second scoring submodule is configured to: determine a reasonable search frequency range based on the information availability level of each assessment item; determine whether the search frequency of each assessment item falls within the corresponding reasonable search frequency range; in response to the search frequency falling within the corresponding reasonable search frequency range, assign a first search efficiency evaluation result to the corresponding assessment item; in response to the search frequency not falling within the corresponding reasonable search frequency range, assign a second search efficiency evaluation result to the corresponding assessment item based on the degree of deviation of the search frequency from the reasonable search frequency range; and determine an efficiency score based on the first or second search efficiency evaluation result of each assessment item.

[0132] In some embodiments, the third scoring submodule is used to: input the answer text into a preset semantic judgment model, so that the semantic judgment model performs keyword matching and sentence pattern matching on the answer text to obtain multiple answer types; determine the answer type ratio corresponding to the information availability level according to each answer type; determine the expected type ratio according to the information availability level; match the answer type ratio with the expected type ratio, and determine the behavior score according to the matching result.

[0133] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0134] The model evaluation device in this embodiment of the disclosure can systematically evaluate the performance of a search-enhanced large language model under different information environments through a complete process from candidate question acquisition, dual-path search query generation, search result verification, information availability level labeling, model search behavior collection, to comprehensive calibration score generation. It can not only evaluate whether the model's final answer is correct, but also further evaluate whether the model searches reasonably, whether it stops invalid searches in a timely manner, whether it refuses to answer when information is unavailable, whether it expresses uncertainty when information is contradictory, and whether its confidence level matches the actual information availability. Simultaneously, it can more comprehensively diagnose the degree of calibration of the search behavior of the search-enhanced large language model, reduce the risk of excessive searching and fabricated answers, and provide quantitative basis for model optimization, system deployment, and reliability assessment.

[0135] This disclosure provides a scenario illustration of a model evaluation method, such as... Figure 6 As shown.

[0136] As previously described, the model evaluation method provided in this disclosure is applied to electronic devices. These electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0137] Specifically, the electronic device may perform the following operations: Multiple candidate assessment questions are obtained. For each candidate assessment question, keywords are extracted from the question text, a first search query is generated based on the keywords, and a second search query is generated based on reference information. The first and second search queries are executed respectively to obtain a first search information set and a second search information set, and search results are generated based on the first and second search information sets. Based on the information consistency between the reference information and the search results, the search results are verified to generate verification results. Based on the verification results, each candidate assessment question is labeled as one of a number of preset information availability levels, resulting in multiple assessment questions. Each assessment question is input into the model being assessed, so that the model can answer each assessment question, obtaining search behavior data and model answer results. Based on the search behavior data and model answer results corresponding to each information availability level, the comprehensive calibration score of the model being assessed is determined, and the search behavior calibration diagnostic result of the model being assessed is generated based on the comprehensive calibration score.

[0138] It should be understood that Figure 6 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 6 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0139] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0140] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0141] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0142] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0143] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0144] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as model evaluation methods. For example, in some embodiments, the model evaluation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the model evaluation method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a model evaluation method by any other suitable means (e.g., by means of firmware).

[0145] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0149] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0150] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0151] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0152] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model evaluation method, comprising: Obtain multiple candidate assessment questions, each candidate assessment question including question text and corresponding reference information; For each candidate assessment question, keywords are extracted from the question text, a first search query is generated based on the keywords, and a second search query is generated based on the reference information. The first search query and the second search query are executed respectively to obtain a first search information set and a second search information set, and search results are generated based on the first search information set and the second search information set; Based on the consistency between the reference information and the search results, the search results are verified, a verification result is generated, and each candidate assessment item is marked as one of a preset multiple information availability levels according to the verification result, thus obtaining multiple assessment items. Each of the aforementioned assessment questions is input into the model being assessed, so that the model being assessed can answer each of the aforementioned assessment questions, thereby obtaining search behavior data and model answer results; Based on the search behavior data and model response results corresponding to each of the information availability levels, a comprehensive calibration score for the evaluated model is determined, and a search behavior calibration diagnostic result for the evaluated model is generated based on the comprehensive calibration score.

2. The method according to claim 1, wherein, The step of executing the first search query and the second search query respectively to obtain a first search information set and a second search information set, and generating search results based on the first search information set and the second search information set, includes: Execute the first search query and the second search query respectively to obtain a first search information set and a second search information set corresponding to the first search query and the second search query respectively; The first K search results are obtained from the first search information set and the second search information set respectively, and a target search set is constructed; where K is an integer not less than 1. The search results are generated by extracting the title, link address, summary text, and sorting position from each search piece of information in the target search set.

3. The method according to claim 1, wherein, After executing the first search query and the second search query respectively to obtain the first search information set and the second search information set, and generating search results based on the first search information set and the second search information set, the method further includes: In response to the candidate assessment questions, the reference information includes at least two contradictory reference claims, and an opposing viewpoint search query is constructed based on the contradictory reference claims; Execute the opposing viewpoint search query to obtain a set of opposing viewpoint search information; Obtain the first K search results from the set of opposing viewpoint search information to construct an opposing viewpoint search set; where K is an integer not less than 1. The title, link address, summary text, and sorting position of each opposing viewpoint search information in the opposing viewpoint search set are extracted and added to the search results.

4. The method according to claim 1, wherein, The step of verifying the search results based on the consistency between the reference information and the search results, and generating a verification result, includes: Extract key reference semantics from the reference information and key result semantics from the search results; The key reference semantics are matched with the key result semantics, and the hit information is determined based on the matching results; the hit information includes at least one of string hit, keyword fragment hit, numerical equivalence hit, and semantic approximation hit; The verification result is generated based on the search results corresponding to the hit information; the verification result includes at least one of the following: search query term, search result sort position, summary text, and link address.

5. The method according to claim 4, wherein, The step of verifying the search results based on the consistency between the reference information and the search results, and generating a verification result, further includes: Extract the claim text corresponding to multiple information sources and the authority level of each information source from the search results; A consistency comparison was performed on the claim texts corresponding to the multiple information sources; In response to the existence of contradictory conclusions, different values, negation relationships, or mutually exclusive conditions in the claim texts corresponding to at least two of the information sources, contradictory information is determined based on the search results; the contradictory information includes at least one of the information source, claim text, link address, and contradiction type. Add the contradictory information and the authority level to the verification result.

6. The method according to claim 1, wherein, The multiple information availability levels include: a first information availability level, a second information availability level, a third information availability level, and a fourth information availability level; The step involves labeling each candidate assessment item with one of a preset multiple information availability levels based on the verification results, resulting in multiple assessment items, including: In response to the verification result indicating that there is contradictory information from at least two information sources in the search results, the corresponding candidate assessment question is marked as a third information availability level; In response to the verification result indicating that there is no contradictory information and the reference information is hit in the search results corresponding to the first search query, the corresponding candidate assessment question is marked as the first information availability level. In response to the verification result indicating that the contradictory information does not exist, the reference information not being found in the search results corresponding to the first search query, and the reference information being found in the search results corresponding to the second search query, the corresponding candidate assessment question is marked as the second information availability level. In response to the verification result indicating that there is no contradictory information and that the search results corresponding to the first search query and the second search query do not match the reference information, the corresponding candidate assessment question is marked as the fourth information availability level. The candidate assessment items that are labeled as the first information availability level, the second information availability level, the third information availability level, or the fourth information availability level will be used as the assessment items.

7. The method according to claim 1, wherein, The step of inputting each of the assessment questions into the model being assessed, so that the model can answer each of the assessment questions, and obtaining search behavior data and model answer results, includes: Based on any of the assessment questions and the preset prompt template, construct assessment prompts; Input the assessment prompts into the model being assessed; Receive the tool call request output by the model being evaluated, and call the search tool according to the tool call request; The search content returned by the search tool is input into the model being evaluated, and the search behavior data is generated based on the tool call request and the search content. The model's response is obtained based on the output of the model being evaluated.

8. The method according to claim 1, wherein, The search behavior data includes at least the number of searches, and the model's response results include at least the response text and confidence level. The step of determining a comprehensive calibration score for the evaluated model based on the search behavior data and model response results corresponding to each of the information availability levels, and generating a search behavior calibration diagnostic result for the evaluated model based on the comprehensive calibration score, includes: The search behavior data and the model answer results are aggregated according to the information availability level to obtain the search behavior data and the model answer results corresponding to each information availability level; For each of the aforementioned information availability levels, the confidence score of the evaluated model is determined based on the corresponding confidence level. For each of the information availability levels, the efficiency score of the evaluated model is determined based on the corresponding number of searches. For each of the aforementioned information availability levels, the corresponding answer text is input into a preset semantic judgment model, so that the semantic judgment model can determine multiple answer types, and the behavior score of the evaluated model is determined based on the proportion of the answer types. The comprehensive calibration score is determined based on the confidence score, the efficiency score, and the behavior score.

9. The method according to claim 8, wherein, For each of the information availability levels, determining the confidence score of the evaluated model based on the corresponding confidence level includes: For each of the aforementioned information availability levels, determine the average confidence level for all model responses. Based on the average confidence level and the reference confidence level corresponding to the information availability level, determine the confidence deviation corresponding to each information availability level; Determine the confidence monotonicity score based on the average confidence level corresponding to each of the aforementioned information availability levels; The confidence score is determined based on the confidence bias and the confidence monotonicity score.

10. The method according to claim 9, wherein, The step of determining the confidence monotonicity score based on the average confidence score corresponding to each of the information availability levels includes: The multiple information availability levels are arranged in descending order of information availability, and the average confidence levels corresponding to adjacent information availability levels are compared. If the average confidence level corresponding to a lower information availability level is not lower than the average confidence level corresponding to a higher information availability level, record a confidence level monotonicity anomaly. The confidence monotonicity score is determined based on the number of confidence monotonicity anomalies.

11. The method according to claim 8, wherein, For each information availability level, the efficiency score of the evaluated model is determined based on the corresponding number of searches, including: Based on the information availability level of each assessment question, determine the corresponding reasonable range of search times; Determine whether the number of searches for each assessment item falls within the corresponding reasonable search range; In response to the search count falling within the corresponding reasonable search count range, a first search efficiency evaluation result is assigned to the corresponding assessment question; In response to the fact that the number of searches does not fall within the corresponding reasonable search number range, a second search efficiency evaluation result is assigned to the corresponding assessment question based on the degree of deviation of the number of searches from the reasonable search number range; The efficiency score is determined based on the first or second search efficiency evaluation results for each of the aforementioned assessment items.

12. The method according to claim 8, wherein, For each information availability level, the corresponding answer text is input into a preset semantic judgment model to determine multiple answer types. Based on the proportion of each answer type, a behavioral score for the evaluated model is determined, including: The answer texts are input into preset semantic determination models, and the semantic determination models perform keyword matching and sentence structure matching on the answer texts to obtain multiple answer types; Based on each of the aforementioned response types, determine the proportion of response types corresponding to the information availability level; The expected type ratio is determined based on the information availability level. The ratio of the response type is matched with the ratio of the expected type, and the behavior score is determined based on the matching result.

13. A model evaluation device, comprising: The question acquisition module is used to acquire multiple candidate assessment questions, wherein the candidate assessment questions include question text and reference information corresponding to the question text; The query construction module is used to extract keywords from the question text for each candidate assessment question, generate a first search query based on the keywords, and generate a second search query based on the reference information. The search execution module is used to execute the first search query and the second search query respectively, obtain the first search information set and the second search information set, and generate search results based on the first search information set and the second search information set; The verification rating module is used to verify the search results based on the information consistency between the reference information and the search results, generate verification results, and mark each candidate assessment item as one of a number of preset information availability levels according to the verification results, thereby obtaining multiple assessment items. The model evaluation module is used to input each of the evaluation questions into the model being evaluated, so that the model being evaluated can answer each of the evaluation questions and obtain search behavior data and model answer results; The result generation module is used to determine the comprehensive calibration score of the evaluated model based on the search behavior data and the model response results corresponding to each of the information availability levels, and to generate the search behavior calibration diagnosis result of the evaluated model based on the comprehensive calibration score.

14. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, Computer instructions are used to cause a computer to perform the method according to any one of claims 1-12.

16. A computer program product comprising a computer program stored on a storage medium, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1-12.

Images