Methods, products, equipment and media for assessing the ability to answer factual knowledge questions

Abstract

The present application discloses a method, product, device and medium for evaluating the ability to answer factual knowledge questions, which relates to the field of natural language processing, including: sampling subgraphs in a historical knowledge graph to generate a factual knowledge question evaluation set; inputting the evaluation set and a priori prompt information group into a pre-trained language model to infer the factual knowledge questions to obtain multiple inference answers; if the multiple inference answers do not contain the preset target characters, then using the a posteriori prompt information to detect the authenticity of the multiple inference answers, if the test results show that the multiple inference answers are consistent with the facts, then the model is judged to have the ability to answer the corresponding questions. This application uses two stages, a priori judgment and a posteriori inspection, to first infer the factual knowledge questions and then perform authenticity detection on the inferred answers. This can prevent the model from misjudging due to hallucinations and improve the efficiency of evaluating the model's ability to answer factual knowledge questions.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to methods, products, devices, and media for evaluating the ability to answer factual knowledge questions. Background Art

[0002] The rapid development of large language models (LLMs) has revolutionized natural language processing (NLP) and artificial intelligence, but it also presents challenges and problems. Through training on vast amounts of text, large language models (applicable in fields such as education, healthcare, finance, law, and content creation) acquire a wealth of factual knowledge. However, they lack the ability to discern whether they know certain facts. For example, when faced with a question, humans can determine whether they are capable of answering it, but large models lack this metacognitive ability. As a result, when a user asks a question, a large language model may give a seemingly reasonable answer that is actually incorrect—a phenomenon known as the large model hallucination.

[0003] To address the above issues, the following two methods have been proposed: one method is to help large language models evaluate their own capabilities by introducing external knowledge. For example, a knowledge base is provided to the model, and the model can determine whether it is capable of answering users' questions by querying the knowledge base. However, this method relies on the knowledge base. If the corresponding content cannot be found in the knowledge base, the model cannot make a correct judgment. The other method is to use example learning to provide the model with some examples with known answers, so that the model can learn to judge whether it can correctly answer similar questions. However, this method does not explicitly introduce metacognitive ability. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, product, device, and medium for evaluating the ability to answer factual knowledge questions, which can improve the efficiency of pre-trained language models in evaluating their ability to answer factual knowledge questions, as well as the model's ability to recognize its own knowledge status. The specific solution is as follows:

[0005] In a first aspect, the present application discloses a method for evaluating the ability to answer factual knowledge questions, comprising:

[0006] Collect the historical knowledge graph of the target domain and sample the subgraphs in the historical knowledge graph to generate a set of factual knowledge question evaluations for the target domain;

[0007] Inputting a factual knowledge question evaluation set and a preset prior prompt information group into a pre-trained language model, reasoning on the factual knowledge questions in the factual knowledge question evaluation set in sequence based on multiple prompt sentences in the prior prompt information group through a linear verification method, and obtaining multiple reasoning answers corresponding to the factual knowledge questions;

[0008] If none of the multiple inference answers contain the preset target character, the multiple inference answers and the preset posterior prompt information are input into the pre-trained language model to detect the authenticity of the multiple inference answers using the posterior prompt information to obtain a detection result; the preset target character is the character that indicates that the inference answer has an anomaly;

[0009] If the test results show that multiple inference answers are consistent with the facts, it is determined that the pre-trained language model has the ability to answer the corresponding factual knowledge questions in the factual knowledge question evaluation set.

[0010] In a second aspect, the present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for assessing the ability to answer factual knowledge questions.

[0011] In a third aspect, the present application discloses an electronic device comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the aforementioned method for assessing the ability to answer factual knowledge questions is implemented.

[0012] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned method for evaluating the ability to answer factual knowledge questions is implemented.

[0013] It can be seen that the present application first collects the historical knowledge graph of the target field and samples the subgraphs in the historical knowledge graph to generate a factual knowledge question evaluation set for the target field, and then inputs the factual knowledge question evaluation set and the preset prior prompt information group into the pre-trained language model, so as to reason the factual knowledge questions in the factual knowledge question evaluation set in sequence through linear verification and based on multiple prompt sentences in the prior prompt information group, and obtain multiple reasoning answers corresponding to the factual knowledge questions; if the multiple reasoning answers do not contain the preset target characters that represent the abnormality of the reasoning answers, then the multiple reasoning answers and the preset posterior prompt information are input into the pre-trained language model to use the posterior prompt information to detect the authenticity of the multiple reasoning answers to obtain the detection results; if the detection results show that the multiple reasoning answers are consistent with the facts, then it is determined that the pre-trained language model has the ability to answer the corresponding factual knowledge questions in the factual knowledge question evaluation set. When evaluating the ability of a pre-trained language model to answer factual knowledge questions, this application is divided into two stages, specifically including a priori judgment and a posteriori inspection. The first stage is to create a factual knowledge question evaluation set based on the historical knowledge graph of a certain field, and then use multiple prompt statements in a preset a priori prompt information group to infer each factual knowledge question in the factual knowledge question evaluation set in turn. The second stage is to input multiple inference answers corresponding to each factual knowledge question obtained by reasoning and preset a posteriori prompt information into the pre-trained language model, so as to detect the authenticity of multiple inference answers. If the test results show that multiple inference answers are consistent with the facts, it is determined that the model has the ability to answer the corresponding factual knowledge questions. Through the above two stages (i.e., the prior judgment stage and the a posteriori check stage), as well as the pre-set prompt information in each stage, not only can the model be guided to judge whether it is capable of answering the corresponding factual knowledge questions, but also through the a posteriori check operation, it can effectively prevent the model from misjudging factual knowledge questions due to hallucinations. In addition, through the above method, the model's ability to answer factual knowledge can be detected without introducing additional model training calculations, thereby improving the efficiency of evaluating the ability to answer factual knowledge questions and the model's ability to recognize its own knowledge status. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0015] Figure 1A flow chart of a method for evaluating the ability to answer factual knowledge questions disclosed in this application;

[0016] Figure 2 A flowchart of the ability to answer a specific factual knowledge question disclosed in this application;

[0017] Figure 3 A flowchart of a method for assessing the ability to answer specific factual knowledge questions disclosed in this application;

[0018] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0021] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0022] The present application discloses a method for evaluating the ability to answer factual knowledge questions. Figure 1 As shown, the method includes:

[0023] Step S11: Collect the historical knowledge graph of the target domain and sample the subgraphs in the historical knowledge graph to generate a factual knowledge question evaluation set for the target domain.

[0024] In this embodiment, first, the knowledge graphs (such as Wikidata, Freebase, etc.) of the fields that need to be evaluated for answering ability (such as education, medical care, finance, law, content creation, etc.) are collected to obtain the historical knowledge graph of the corresponding field, and then the subgraphs in the historical knowledge graph of the field are sampled to generate a factual knowledge question evaluation set for the field. As can be seen from the above, the factual knowledge question evaluation set in this application is created in real time and dynamically based on the historical knowledge graph. Compared with directly using the existing factual knowledge question evaluation set, it can ask more accurate and targeted questions to the corresponding field. Among them, the knowledge graph (KG) of any field must contain entities, relationships and other content.

[0025] Specifically, sampling subgraphs in the historical knowledge graph to generate an evaluation set of factual knowledge questions for the target domain can include: sampling subgraphs in the historical knowledge graph based on different sampling rules to obtain multiple subgraph sets; merging the multiple subgraph sets to obtain an evaluation subgraph set, and inputting the evaluation subgraph set into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the evaluation subgraph set. That is, when sampling subgraphs in the historical knowledge graph, sampling can be performed based on different pre-set sampling rules to obtain subgraph sets under each sampling rule, and then the subgraph sets under all sampling rules are merged, and the evaluation subgraph set obtained after the merger is input into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the entire evaluation subgraph set.

[0026] In a specific embodiment, subgraphs in the historical knowledge graph are sampled based on different sampling rules to obtain multiple subgraph sets, which may specifically include: sampling subgraphs in the historical knowledge graph based on a high-frequency knowledge sampling strategy and a long-tail knowledge sampling strategy to obtain a high-frequency subgraph set and a long-tail subgraph set; wherein, the high-frequency knowledge sampling strategy is a sampling strategy that uses a preset number of nodes ranked before the node degree of each node in the knowledge graph as high-frequency knowledge nodes; the long-tail knowledge sampling strategy is a sampling strategy that uses nodes in the subgraph of the knowledge graph whose subgraph entropy value is greater than the preset entropy value as long-tail knowledge nodes. In this embodiment, in order to ensure coverage of the knowledge graph, high-frequency knowledge and long-tail knowledge are considered simultaneously when sampling subgraphs in the historical knowledge graph. Specifically, the high-frequency knowledge sampling strategy can be used to sample subgraphs in the historical knowledge graph to obtain the corresponding high-frequency subgraph set. , where the high-frequency knowledge sampling strategy specifically uses a preset number of nodes (such as the top K nodes, with K=1000 by default) ranked by the node degree of each node in the knowledge graph as the sampling strategy for high-frequency knowledge nodes. In other words, the high-frequency knowledge sampling strategy is based on the degree of the knowledge node. The higher the degree, the stronger the association of the node in the knowledge graph, representing high-frequency knowledge. The calculation formula for the node degree is:

[0027] ;

[0028] Where, is the degree of node v, which indicates the total number of edges (i.e., relationships) associated with the node in the knowledge graph; is the set of edges in the knowledge graph, each edge represents the relationship between two nodes (such as "Li Bai-Dynasty-Tang Dynasty"); is an indicator function, which means that the value is 1 when the node v belongs to the edge e, otherwise it is 0.

[0029] Then, based on the long-tail knowledge sampling strategy, the subgraphs in the historical knowledge graph are sampled to obtain the corresponding long-tail subgraph set ; Among them, the long-tail knowledge sampling strategy is to sample nodes in the subgraph of the knowledge graph whose subgraph entropy value is greater than the preset entropy value as long-tail knowledge nodes. In other words, the long-tail knowledge sampling strategy is based on graph entropy. The purpose of this strategy is to capture low-frequency but critical diversified knowledge. Subgraphs with high entropy values ​​contain more low-frequency but diversified nodes, representing long-tail knowledge. Among them, the calculation formula of the subgraph entropy value is:

[0030] ;

[0031] Where, The entropy value of the subgraph S is used to measure the diversity of node distribution. The larger the entropy value, the more dispersed the node distribution. represents the probability of node v appearing in subgraph S, and the calculation formula is: .

[0032] The long-tail knowledge sampling strategy is to select the subgraph entropy value Low-degree nodes, where the entropy value is preset It can be obtained based on experimental verification and can be adjusted according to actual application needs. It is used to filter long-tail knowledge nodes and can be set to 2.5 by default.

[0033] It's important to note that the high-frequency knowledge sampling strategy is used to filter highly correlated core nodes in the knowledge graph (such as "M scientist"), thereby covering commonsense knowledge. The long-tail knowledge sampling strategy is used to capture low-frequency but critical knowledge nodes (such as "quantum entanglement experimental device"), which can improve the comprehensiveness of the evaluation. By sampling subgraphs from a domain's knowledge graph using a hybrid sampling strategy based on high-frequency and long-tail knowledge sampling strategies, we can ensure coverage of the knowledge graph, obtaining sets of high-frequency and long-tail subgraphs. This in turn generates an evaluation set of factual knowledge questions covering both high-frequency and long-tail knowledge. This addresses the coverage bias issue in static evaluation sets and facilitates the accuracy of subsequent model assessments of their ability to answer factual knowledge questions.

[0034] Specifically, multiple subgraph sets are merged to obtain an evaluation subgraph set, and the evaluation subgraph set is input into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the evaluation subgraph set. This may include: merging a high-frequency subgraph set and a long-tail subgraph set to obtain an evaluation subgraph set, and inputting the evaluation subgraph set into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the evaluation subgraph set. In this embodiment, the high-frequency subgraph set corresponding to the target domain is first and long-tail subgraph collections Merge to get the evaluation subgraph set , and then the evaluation subgraph set Input into the pre-created factual knowledge question generation template to construct a set of subgraphs that can cover the entire evaluation The factual knowledge question evaluation set contains multiple factual knowledge questions, which can evaluate the ability of the pre-trained language model to answer factual knowledge questions in a general knowledge domain or a specific knowledge domain (depending on the subject and content of the knowledge graph used).

[0035] It should be pointed out that the types of factual knowledge question generation templates specifically include direct question type, triple mask type and path query type; among them, the direct question type is the type that directly generates questions based on triples in the knowledge graph; the triple mask type is the type that generates fill-in-the-blank questions by randomly masking the terminal entity or the tail entity; the path query type is the type that extracts a relational path with a length less than or equal to a preset length, and generates multi-hop reasoning questions based on the relational path. Specifically, the direct question type generates the question "Which dynasty is h in?" for the triple (h, r, t). For example, when the triple (h, r, t) is (Li Bai, dynasty, Tang Dynasty), the generated question is "Which dynasty did Li Bai live in?" The masked triple type generates fill-in-the-blank questions by randomly masking the first or last entity h or t. For example, when the triple (h, r, t) is (Li Bai, dynasty, Tang Dynasty), the generated question is "The dynasty that ____ lived in is the Tang Dynasty." The path query type extracts relational paths of length ≤ 3 and generates multi-hop reasoning questions based on these relational paths, such as "Li Bai → "Viewing the Waterfall at Mount Lu" → What dynasty was the author of the poem?" By pre-setting multiple factual knowledge question generation templates, different types of factual knowledge questions can be generated, which facilitates a more comprehensive evaluation of the model's ability to answer factual knowledge questions, thereby improving the accuracy of the model's ability to answer factual knowledge questions.

[0036] In this embodiment, the evaluation subgraph set is input into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the evaluation subgraph set. Specifically, it may include: inputting the evaluation subgraph set into a pre-created factual knowledge question generation template containing direct question type, triple mask type and path query type, to generate questions corresponding to the direct question type, triple mask type and path query type in accordance with a preset generation ratio, and obtaining an evaluation set of factual knowledge questions covering the evaluation subgraph set; wherein, the number of questions generated by each subgraph in the evaluation subgraph set is determined based on the number of nodes of the corresponding subgraph, and the ratio of the number of questions generated by the high-frequency subgraph set to the number of questions generated by the long-tail subgraph set is a preset value. In this embodiment, when generating a covering evaluation subgraph set based on the factual knowledge question generation template, When evaluating the factual knowledge question set, factual knowledge questions corresponding to different template types can be generated according to a preset ratio. For example, a corresponding number of factual knowledge questions can be generated according to a ratio of 2:4:4 for direct question type, triple mask type, and path query type. In addition, the number of questions generated for each subgraph in the evaluation subgraph set is determined based on the number of nodes in the corresponding subgraph, and the high-frequency subgraph set The number of generated questions and the set of long-tail subgraphs The ratio of the number of generated questions is a preset value. For example, when the number of nodes in a subgraph in the evaluation subgraph set is When , the number of questions generated by this subgraph is , and the generated problem distribution is: high-frequency subgraph set The generated questions account for 70% of the total generated questions, and the long-tail subgraph collection The generated problems account for 30% of the total generated problems, that is, the high-frequency subgraph set Generated Problems and Long-tail Subgraph Collections The ratio of generated questions is 7:3. This application pre-designs templates for generating different types of factual knowledge questions by combining the structural characteristics of the knowledge graph, and automatically constructs factual knowledge questions that can well cover the subgraph set and according to the preset proportional relationship, thereby more reasonably and comprehensively generating a factual knowledge question evaluation set for a certain field, which is conducive to the subsequent model to conduct a more comprehensive evaluation of factual knowledge questions in the corresponding field, thereby improving the accuracy of the model's ability to answer factual knowledge questions in the field.

[0037] Step S12: Input the factual knowledge question evaluation set and the preset prior prompt information group into the pre-trained language model, and use linear verification to infer the factual knowledge questions in the factual knowledge question evaluation set in sequence based on multiple prompt sentences in the prior prompt information group to obtain multiple inference answers corresponding to the factual knowledge questions.

[0038] In this embodiment, after sampling the subgraphs in the historical knowledge graph to obtain the factual knowledge question evaluation set for the target field, the factual knowledge questions in the factual knowledge question evaluation set and the preset prior prompt information group can be input into the pretrained language model (PLM) together, so as to use a linear verification method and based on multiple prompt sentences in the prior prompt information group to sequentially infer each factual knowledge question in the input factual knowledge question evaluation set, thereby obtaining multiple reasoning answers corresponding to a single factual knowledge question; wherein the pretrained language model is a large language model (LLM) related to the target field corresponding to the factual knowledge question evaluation set, such as a medical question and answer large model, a financial large model, a government affairs large model, etc., and the model can adopt a Transformer network structure.

[0039] In addition, the number of multiple reasoning answers corresponding to a single factual knowledge question is consistent with the number of multiple prompt statements in the prior prompt information group. This application does not make specific restrictions on the number of prompt statements in the prior prompt information group, and it can be set according to actual application requirements.

[0040] For example, various factual knowledge questions and preset prompt information groups (prompt groups) are input into the medical question-answering big model, which then determines whether the participant knows the factual knowledge questions related to a specific medical field. The prompt information group (prompt group) specifically includes the following four prompt statements: 1. Do you know the answer to the following question? If not, please reply "I don't know." 2. Are you capable of answering the following question? If not, please reply "I don't know." 3. Do you have any knowledge of the following question? If not, please reply "I don't know." 4. Do you think you have enough wisdom to answer the following question? If not, please reply "I don't know." It should be noted that the model uses a linear verification method (which simulates the stress testing scenario in human conversation) during reasoning. That is, for question q, the model uses the above four prior prompt statements one by one to perform reasoning. For example, when the reasoning result corresponding to the first prior prompt statement does not contain the negative answer character "I don't know", that is, the reasoning answer corresponding to the first prior prompt statement is answered normally, then the next prior prompt statement is asked again until all four prior prompt statements are called. If all four prior prompt statements are answered normally, that is, none of them contain the negative answer character "I don't know", then question q is considered to pass the prior judgment. It should be noted that the number of prior prompt statements in the prior prompt information group is not necessarily better, because too many prompt statements will affect the accuracy of the prior judgment and affect processing efficiency. The default number of prompt statements can be 4.

[0041] It should be pointed out that the process of reasoning about the factual knowledge questions in the factual knowledge question evaluation set in sequence through a linear verification method and based on multiple prompt statements in the prior prompt information group may also include: if the current reasoning answer contains preset target characters, then suspend the current reasoning operation, and directly determine that the pre-trained language model does not have the ability to answer the corresponding factual knowledge questions in the factual knowledge question evaluation set. That is, in the process of reasoning about a single factual knowledge question in sequence through multiple prompt statements (Prompt, used to prompt the user to input text or instructions) in the prior prompt information group, if it is detected that the current reasoning answer contains preset target characters, such as "I don't know", then immediately stop the current reasoning operation, and at the same time determine that the pre-trained language model does not have the ability to answer the corresponding factual knowledge questions in the factual knowledge question evaluation set. The model does not have the ability to answer the corresponding factual knowledge question q, that is, when the answer to any prompt sentence is "I don't know", it is considered that the model The model does not have the ability to answer the factual knowledge question. Immediately terminating the current reasoning operation when detecting that the preset target character is included in the reasoning answer can avoid subsequent invalid reasoning judgments, thereby improving the reasoning efficiency of the model.

[0042] In this embodiment, in order to more accurately and quickly reason about the factual knowledge questions in the factual knowledge question evaluation set, the multiple prompt statements in the prior prompt information group can also be sorted before reasoning, such as sorting the multiple prompt statements in the prompt information group according to the sentence length, content complexity, number of prompt verbs contained in the prompt statement, and then using the sorted multiple prompt statements to reason about the factual knowledge questions in the factual knowledge question evaluation set, thereby obtaining multiple reasoning answers to the corresponding factual knowledge questions. In addition, there can also be an association relationship between different prompt statements, such as prompt statement a is a combination of prompt statement b and prompt statement c. Through the above-mentioned diversified prompt statement design and sorting, the accuracy of the model's reasoning on factual knowledge questions can be further improved.

[0043] Step S13: If none of the multiple inference answers contain the preset target character, the multiple inference answers and the preset posterior prompt information are input into the pre-trained language model to use the posterior prompt information to detect the authenticity of the multiple inference answers to obtain the detection results; the preset target character is the character that represents the abnormality of the inference answer.

[0044] In this embodiment, if none of the multiple reasoning answers corresponding to a factual knowledge question q contains a preset target character, the preset target character is a character that indicates that the reasoning answer is abnormal (such as "I don't know"), then the multiple reasoning answers and the preset posterior prompt information are input into the pre-trained language model. In this case, the authenticity of multiple reasoning answers corresponding to a factual knowledge question q is tested using a posteriori prompt information to obtain a test result corresponding to the factual knowledge question q. For example, the authenticity of multiple reasoning answers corresponding to a factual knowledge question q is tested using a posteriori prompt information (Is the following text factually correct? If not, please answer "Factually incorrect"), and if the test result contains the characters "Factually incorrect", the multiple reasoning answers corresponding to the factual knowledge question q are judged to be inconsistent with the facts; if the test result does not contain the characters "Factually incorrect", the multiple reasoning answers corresponding to the factual knowledge question q are judged to be consistent with the facts.

[0045] In a specific embodiment, multiple reasoning answers and preset a posteriori prompt information are input into a pre-trained language model to detect the authenticity of the multiple reasoning answers using the a posteriori prompt information to obtain a detection result. Specifically, it may include: inputting multiple reasoning answers and preset first a posteriori prompt information into the pre-trained language model to integrate the multiple reasoning answers into a text based on the first a posteriori prompt information to obtain an integrated text; inputting the integrated text and preset second a posteriori prompt information into the pre-trained language model to detect the authenticity of the integrated text based on the second a posteriori prompt information to obtain a detection result. In this embodiment, in order to overcome the hallucination problem of the pre-trained language model, that is, the inability to make accurate answers to certain factual knowledge questions, a posteriori check is further performed on the basis of a priori judgment, thereby avoiding the model from giving incorrect answers to factual knowledge questions due to hallucinations. For details, see Figure 2 As shown, prior judgment specifically refers to the process of linear reasoning on a factual knowledge question q input through a large language model and based on a preset prior prompt information group. If the prior prompt information group contains 4 prompt statements, 4 corresponding reasoning answers will be obtained, which can be recorded as If any of the inference answers contains a specific character (such as "I don't know"), it is directly determined that the large language model does not have the ability to answer the question q. If none of the four inference answers contain a specific character (such as "I don't know"), it means that the prior judgment has been passed and a posteriori check can be performed; see Figure 2 As shown, the posterior check specifically refers to the authenticity detection of multiple inference answers through a large language model and based on two preset posterior prompt information. Specifically, the multiple inference answers can be combined into a text using a large language model and based on the first posterior prompt information (for example, "Please combine the following answers into a text: {answer list}"). Integrate into a piece of text to obtain an integrated text A; then, input the integrated text A and the second a posteriori prompt information (for example, are the following text facts true and correct? If not, please answer "the facts are wrong") into the large language model to detect the authenticity of the integrated text A and obtain the corresponding test results. This application uses two stages, a priori judgment and a posteriori inspection, to not only reason about factual knowledge questions, but also to detect the authenticity of the reasoning answers obtained after reasoning, thereby effectively avoiding the large model from giving incorrect answers to factual knowledge questions due to hallucinations, and preventing the large model from misjudging factual knowledge questions due to hallucinations. In addition, combining multiple answers to a priori judgments (i.e., multiple reasoning answers) into a coherent text and verifying the authenticity of the answers through a posteriori prompt information can reduce the risk of misjudgment caused by the fragmentation of answers (i.e., reasoning answers).

[0046] Furthermore, after the authenticity of the integrated text is tested based on the second a posteriori prompt information and the test result is obtained, the following process may be further performed: if the test result shows that the integrated text is inconsistent with the facts, then it is determined that the pre-trained language model does not have the ability to answer the corresponding factual knowledge questions in the factual knowledge question evaluation set. For example, see Figure 2 As shown in Figure 1, if the detection result contains characters that indicate “factual error”, it is directly determined that the large language model does not have the ability to answer the corresponding factual knowledge question q.

[0047] Step S14: If the detection result shows that multiple inference answers are consistent with the facts, it is determined that the pre-trained language model has the ability to answer the corresponding factual knowledge questions in the factual knowledge question evaluation set.

[0048] In this embodiment, if the test results show that multiple inference answers are consistent with the facts, for example, the test results do not contain specific characters of "factual error", it indicates that the posterior check has passed. At this time, it can be determined that the large language model has the ability to answer the corresponding factual knowledge question q.

[0049] It can be seen that the embodiment of the present application is divided into two stages when evaluating the ability of the pre-trained language model to answer factual knowledge questions, specifically including two stages: a priori judgment and a posteriori inspection. The first stage is to create a factual knowledge question evaluation set based on the historical knowledge graph of a certain field, and then use multiple prompt statements in the preset a priori prompt information group to infer each factual knowledge question in the factual knowledge question evaluation set in turn. The second stage is to input multiple reasoning answers corresponding to each factual knowledge question obtained by reasoning and the preset a posteriori prompt information into the pre-trained language model, so as to detect the authenticity of the multiple reasoning answers. If the test results show that the multiple reasoning answers are consistent with the facts, it is determined that the model has the ability to answer the corresponding factual knowledge questions. Through the above two stages (i.e., the prior judgment stage and the a posteriori check stage), as well as the pre-set prompt information in each stage, not only can the model be guided to judge whether it is capable of answering the corresponding factual knowledge questions, but also through the a posteriori check operation, it can effectively prevent the model from misjudging factual knowledge questions due to hallucinations. In addition, through the above method, the model's ability to answer factual knowledge can be detected without introducing additional model training calculations, thereby improving the efficiency of evaluating the ability to answer factual knowledge questions and the model's ability to recognize its own knowledge status.

[0050] The present application embodiment discloses a specific method for evaluating the ability to answer factual knowledge questions. Figure 3 As shown, the method includes:

[0051] Step S21: Collect the historical knowledge graph of the target field, and sample the subgraphs in the historical knowledge graph based on different sampling rules to obtain multiple subgraph sets.

[0052] Step S22: merging multiple subgraph sets to obtain an evaluation subgraph set, and inputting the evaluation subgraph set into a pre-created factual knowledge question generation template to construct a factual knowledge question evaluation set covering the evaluation subgraph set.

[0053] Step S23: Input the factual knowledge question evaluation set and the preset prior prompt information group into the pre-trained language model, and use linear verification to infer the factual knowledge questions in the factual knowledge question evaluation set in sequence based on multiple prompt statements in the prior prompt information group to obtain multiple inference answers corresponding to the factual knowledge questions.

[0054] Step S24: If none of the multiple inference answers contain the preset target character, the multiple inference answers and the preset posterior prompt information are input into the pre-trained language model to use the posterior prompt information to detect the authenticity of the multiple inference answers to obtain the detection results; the preset target character is the character that represents the abnormality of the inference answer.

[0055] Step S25: If the detection result shows that multiple inference answers are consistent with the facts, it is determined that the pre-trained language model has the ability to answer the corresponding factual knowledge questions in the factual knowledge question evaluation set.

[0056] Step S26: Based on the ability of the pre-trained language model to answer factual knowledge questions and the preset composite evaluation indicators, the accuracy of the pre-trained language model's answers and the knowledge coverage are counted to obtain the model answer accuracy and model knowledge coverage.

[0057] In this embodiment, after sequentially reasoning and judging each factual knowledge question in the factual knowledge question evaluation set and testing the authenticity of the reasoned answers, a comprehensive evaluation of the pre-trained language model's ability to answer factual knowledge questions in the target domain can be conducted based on the multiple reasoned answers corresponding to all factual knowledge questions in the factual knowledge question evaluation set (i.e., the results of the a priori judgment) and the test results (i.e., the results of the a posteriori verification). Specifically, based on the pre-trained language model's ability to answer each factual knowledge question q in the factual knowledge question evaluation set and preset composite evaluation indicators (including accuracy and coverage), the accuracy and knowledge coverage of the pre-trained language model's answers can be statistically analyzed to obtain the corresponding model answer accuracy (Accuracy) and model knowledge coverage (Coverage).

[0058] Specifically, based on the ability of the pre-trained language model to answer factual knowledge questions in the factual knowledge question evaluation set and the preset composite evaluation indicators, the accuracy and knowledge coverage of the pre-trained language model's answers are statistically analyzed to obtain the model answer accuracy and model knowledge coverage, which may include: screening a first category of factual knowledge questions from the factual knowledge question evaluation set; the first category of factual knowledge questions are factual knowledge questions for which none of the corresponding multiple reasoning answers contain preset target characters; screening a second category of factual knowledge questions from the first category of factual knowledge questions; the second category of factual knowledge questions are first category factual knowledge questions for which the test results represent that the corresponding multiple reasoning answers are consistent with the facts; calculating the ratio of the number of questions of the second category of factual knowledge questions to the total number of factual knowledge questions in the factual knowledge question evaluation set to obtain the model answer accuracy of the pre-trained language model; counting the number of target nodes in the evaluation subgraph set; the target node is a node for which none of the corresponding multiple reasoning answers contain preset target characters; calculating the ratio of the number of target nodes to the total number of nodes in the evaluation subgraph set to obtain the model knowledge coverage of the pre-trained language model. In this embodiment, the first type of factual knowledge questions can be screened out from the factual knowledge question evaluation set. Specifically, the first type of factual knowledge questions are factual knowledge questions for which none of the corresponding multiple reasoning answers contain the preset target characters (such as "I don't know"), that is, Figure 2 Then, the second type of factual knowledge questions are selected from the first type of factual knowledge questions. The second type of factual knowledge questions are the first type of factual knowledge questions whose test results represent that the corresponding multiple reasoning answers are consistent with the facts, that is, Figure 2 The factual knowledge questions q that pass the posterior check are further calculated by calculating the ratio of the number of the second type of factual knowledge questions to the total number of all factual knowledge questions in the factual knowledge question evaluation set, thereby obtaining the model’s answer accuracy. The specific calculation formula is:

[0059] ;

[0060] Where, represents the number of factual knowledge questions that pass both a priori judgment and a posteriori verification. Represents the total number of questions in the factual knowledge question set. The model's accuracy rate reflects the accuracy of the model's responses to known questions. A higher value indicates greater reliability of the model's knowledge.

[0061] Next, the statistical evaluation subgraph set The target nodes are nodes that do not contain the preset target characters (such as "I don't know") in the corresponding multiple reasoning answers, and the ratio of the number of target nodes to the number of all nodes in the evaluation subgraph set is calculated to obtain the model knowledge coverage. The specific calculation formula is:

[0062] ;

[0063] Where, Indicates the number of nodes that do not contain the preset target characters in the corresponding multiple reasoning answers, that is, the evaluation subgraph The number of nodes whose questions are answered correctly; Represents the evaluation subgraph The total number of all nodes in .

[0064] Step S27: Based on the model answer accuracy and model knowledge coverage, a comprehensive evaluation is performed on the ability of the pre-trained language model to answer factual knowledge questions in the target domain to obtain a performance evaluation result.

[0065] In this embodiment, after calculating the answer accuracy and knowledge coverage of the pre-trained language model, the pre-trained language model's ability to answer factual knowledge in the target field can be comprehensively evaluated directly based on the calculated model answer accuracy and model knowledge coverage to obtain corresponding performance evaluation results.

[0066] Specifically, based on the model answer accuracy and model knowledge coverage, a comprehensive evaluation is performed on the ability of the pre-trained language model to answer factual knowledge questions in the target field to obtain a performance evaluation result, which may include: calculating the product of the model answer accuracy and the first weight coefficient to obtain a first calculation result, and calculating the product of the model knowledge coverage and the second weight coefficient to obtain a second calculation result; and the sum of the first weight coefficient and the second weight coefficient is 1; calculating the sum of the first calculation result and the second calculation result to obtain the performance evaluation result of the pre-trained language model. In this embodiment, the calculation formula of the performance evaluation result can be specifically expressed as:

[0067] ;

[0068] Where, It represents the first weight coefficient, that is, the weight coefficient of the accuracy rate. The value of this coefficient can reflect the importance attached to accuracy. represents the second weight coefficient, namely the weight coefficient of coverage, the value of which can reflect the degree of attention paid to the comprehensiveness of knowledge. In a specific embodiment, and , and meet , represents the weight normalization constraint, ensuring that the comprehensive score (i.e., performance evaluation result) is in the range [0,1].

[0069] The comprehensive performance of the model is evaluated by the model's answer accuracy, model knowledge coverage and corresponding weight coefficients, which can not only balance accuracy and comprehensiveness, but also adjust according to needs. The value of size to emphasize the coverage of knowledge.

[0070] For more specific processing procedures of steps S21 to S25 , reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be elaborated here.

[0071] As can be seen, the embodiment of the present application evaluates the ability of the pre-trained language model to answer factual knowledge questions through a two-stage strategy of prior judgment (verifying whether the model "knows") and a posteriori check (verifying the authenticity of the answer). It can accurately determine whether the answer is correct, solve the problem of misjudgment caused by model hallucination, and reduce the risk of misjudgment caused by model hallucination. In addition, the embodiment of the present application introduces a set of prompt statements in the prior judgment stage to guide the model to determine whether it is capable of answering factual knowledge questions, and adds a posteriori check operation after the model answers, thereby preventing the model from misjudging factual knowledge due to hallucination. Through this strategy, the factual knowledge ability of the model can be detected without introducing the computational complexity of additional model training, providing a basis for applications such as preventing large model hallucinations and using external knowledge to strengthen models. In addition, the embodiment of the present application can quantify the comprehensiveness of the model's knowledge mastery through a composite evaluation indicator including accuracy (used to count the proportion of questions verified in the two-stage verification, measuring the accuracy of the model's answers) and coverage (used to count the proportion of knowledge nodes answered correctly, measuring the scope of model knowledge coverage), and solve the limitations of traditional single indicators.

[0072] For example, a knowledge graph in the scientific field (such as Wikidata) is collected, and then the "Physicist" subgraph within the knowledge graph is sampled. The resulting evaluation subgraph set is then fed into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the entire evaluation subgraph set. This evaluation set includes multiple factual knowledge questions, such as the high-frequency knowledge question "What is the nationality of scientist M?" (degree = 120) and the long-tail knowledge question "The ____ experiment measured the charge of an electron for the first time" (entropy = 3.2). Each question in the evaluation set is then subjected to a priori judgment and a posteriori check. The specific process is: Question "The ____ experiment measured the charge of an electron for the first time" → Pass the entity check → Pass the four prompt verifications → Comprehensive reasoning answer "The oil drop experiment" → Pass the a posteriori check. Furthermore, the entire factual knowledge question evaluation set was evaluated. The specific process was: Accuracy = 87% → Coverage = 80% → Score = 0.7 × 87% + 0.3 × 80% = 84.9%. That is, the comprehensive evaluation score of the model's ability to answer factual knowledge in the scientific field was 84.9%.

[0073] The embodiments of the present application also provide a device for assessing the ability to answer factual knowledge questions. For descriptions of the features in the embodiments corresponding to the device for assessing the ability to answer, please refer to the relevant descriptions of the embodiments corresponding to the method for assessing the ability to answer, and will not be repeated here.

[0074] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0075] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the method for assessing the ability to answer factual knowledge questions disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0076] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0077] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0078] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, and can be Windows Server, NetWare, Unix, Linux, etc. In addition to including a computer program capable of implementing the method for assessing the ability to answer factual knowledge questions performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0079] Furthermore, this application discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned method for assessing the ability to answer factual knowledge questions. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be further described here.

[0080] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for assessing the ability to answer factual knowledge questions disclosed above.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0082] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0084] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0085] The above is a detailed introduction to the method, product, equipment and medium for assessing the ability to answer factual knowledge questions provided by this application. Specific examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core ideas. At the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A method for evaluating the ability to answer factual knowledge questions, characterized in that: include: Collecting a historical knowledge graph of a target domain and sampling subgraphs in the historical knowledge graph to generate a factual knowledge question evaluation set for the target domain; The factual knowledge question evaluation set and a preset priori prompt information group are input into a pre-trained language model, and the factual knowledge questions in the factual knowledge question evaluation set are sequentially inferred based on multiple prompt statements in the priori prompt information group through a linear verification method to obtain multiple inference answers corresponding to the factual knowledge questions; the linear verification method refers to a method of sequentially inferring the factual knowledge questions one by one using the priori prompt statements in a stress test scenario in a simulated conversation; the multiple prompt statements in the priori prompt information group are texts or instructions used to prompt the user to input; If none of the multiple inference answers contain the preset target character, inputting the multiple inference answers and the preset a posteriori prompt information into the pre-trained language model, so as to detect the authenticity of the multiple inference answers using the a posteriori prompt information and obtain a detection result; The preset target character is a character indicating that the reasoning answer is abnormal; If the detection result shows that the multiple inference answers are consistent with the facts, it is determined that the pre-trained language model has the ability to answer the corresponding factual knowledge questions in the factual knowledge question assessment set; The sampling of subgraphs in the historical knowledge graph to generate an evaluation set of factual knowledge questions for the target domain includes: sampling subgraphs in the historical knowledge graph based on different sampling rules to obtain multiple subgraph sets; merging the multiple subgraph sets to obtain an evaluation subgraph set, and inputting the evaluation subgraph set into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the evaluation subgraph set; The subgraphs in the historical knowledge graph are sampled based on different sampling rules to obtain multiple subgraph sets, including: sampling the subgraphs in the historical knowledge graph based on a high-frequency knowledge sampling strategy and a long-tail knowledge sampling strategy to obtain a high-frequency subgraph set and a long-tail subgraph set; wherein the high-frequency knowledge sampling strategy is a sampling strategy that uses a preset number of nodes ranked before the node degree of each node in the knowledge graph as high-frequency knowledge nodes; the long-tail knowledge sampling strategy is a sampling strategy that uses nodes in the subgraph of the knowledge graph whose subgraph entropy value is greater than a preset entropy value as long-tail knowledge nodes; the long-tail knowledge sampling strategy is based on graph entropy and is used to capture low-frequency but critical diversified knowledge. A subgraph with a high entropy value contains more low-frequency but diversified nodes, representing long-tail knowledge; The merging of the multiple subgraph sets to obtain an evaluation subgraph set, and inputting the evaluation subgraph set into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the evaluation subgraph set, comprising: merging the high-frequency subgraph set and the long-tail subgraph set to obtain an evaluation subgraph set, and inputting the evaluation subgraph set into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the evaluation subgraph set; The types of factual knowledge question generation templates include direct question type, triple mask type and path query type; wherein the direct question type is a type that directly generates questions based on triples in the knowledge graph; the triple mask type is a type that generates fill-in-the-blank questions by randomly masking the terminal entity or the tail entity; the path query type is a type that extracts a relational path with a length less than or equal to a preset length and generates a multi-hop reasoning question based on the relational path; The step of inputting the evaluation subgraph set into a pre-created factual knowledge question generation template to construct an evaluation set of factual knowledge questions covering the evaluation subgraph set comprises: inputting the evaluation subgraph set into a pre-created factual knowledge question generation template comprising a direct question type, a triple mask type and a path query type, to generate questions corresponding to the direct question type, the triple mask type and the path query type in accordance with a preset generation ratio, and obtaining an evaluation set of factual knowledge questions covering the evaluation subgraph set; wherein the number of questions generated for each subgraph in the evaluation subgraph set is determined based on the number of nodes of the corresponding subgraph, and the ratio of the number of questions generated by the high-frequency subgraph set to the number of questions generated by the long-tail subgraph set is a preset value; The step of inputting the multiple inference answers and preset a posteriori prompt information into the pre-trained language model to detect the authenticity of the multiple inference answers using the a posteriori prompt information to obtain a detection result includes: inputting the multiple inference answers and preset first a posteriori prompt information into the pre-trained language model to integrate the multiple inference answers into a piece of text based on the first a posteriori prompt information to obtain an integrated text; and inputting the integrated text and preset second a posteriori prompt information into the pre-trained language model to detect the authenticity of the integrated text based on the second a posteriori prompt information to obtain a detection result.

2. The method for evaluating the ability to answer factual knowledge questions according to claim 1, characterized in that: The process of sequentially reasoning the factual knowledge questions in the factual knowledge question evaluation set in a linear verification manner based on the multiple prompt statements in the priori prompt information group also includes: If the current reasoning answer contains the preset target character, the current reasoning operation is suspended, and it is directly determined that the pre-trained language model does not have the ability to answer the corresponding factual knowledge question in the factual knowledge question evaluation set.

3. The method for evaluating the ability to answer factual knowledge questions according to claim 1, characterized in that: After determining that the pre-trained language model is capable of answering corresponding factual knowledge questions in the factual knowledge question evaluation set, the method further includes: Based on the ability of the pre-trained language model to answer the factual knowledge questions in the factual knowledge question evaluation set and the preset composite evaluation indicators, the accuracy and knowledge coverage of the pre-trained language model's answers are counted to obtain the model answer accuracy rate and the model knowledge coverage rate; Based on the model's answer accuracy and model knowledge coverage, a comprehensive evaluation is performed on the ability of the pre-trained language model to answer factual knowledge questions in the target domain to obtain a performance evaluation result.

4. The method for evaluating the ability to answer factual knowledge questions according to claim 3, characterized in that: The ability of the pre-trained language model to answer the factual knowledge questions in the factual knowledge question evaluation set and the preset composite evaluation indicators are used to perform statistics on the accuracy and knowledge coverage of the pre-trained language model's answers to obtain the model answer accuracy rate and model knowledge coverage rate, including: Selecting a first category of factual knowledge questions from the factual knowledge question evaluation set; the first category of factual knowledge questions being factual knowledge questions for which none of the corresponding multiple reasoning answers contain the preset target character; Screening a second category of factual knowledge questions from the first category of factual knowledge questions; the second category of factual knowledge questions being the first category of factual knowledge questions for which the test result represents that the corresponding multiple reasoning answers are consistent with the facts; Calculating the ratio of the number of the second-category factual knowledge questions to the total number of factual knowledge questions in the factual knowledge question evaluation set to obtain the model answer accuracy of the pre-trained language model; Counting the number of target nodes in the evaluation subgraph set; the target node is a node that does not contain the preset target character in the corresponding multiple reasoning answers; The ratio of the number of target nodes to the total number of nodes in the evaluation subgraph set is calculated to obtain the model knowledge coverage of the pre-trained language model.

5. The method for evaluating the ability to answer factual knowledge questions according to claim 3, characterized in that: The ability of the pre-trained language model to answer factual knowledge questions in the target domain is comprehensively evaluated based on the model's answer accuracy and model knowledge coverage, to obtain a performance evaluation result, including: Calculating the product of the model's answer accuracy and a first weight coefficient to obtain a first calculation result, and calculating the product of the model's knowledge coverage and a second weight coefficient to obtain a second calculation result; the sum of the first weight coefficient and the second weight coefficient is 1; Calculate the sum of the first calculation result and the second calculation result to obtain a performance evaluation result of the pre-trained language model.

6. The method for evaluating the ability to answer factual knowledge questions according to claim 1, characterized in that: After the authenticity of the integrated text is detected based on the second a posteriori prompt information and a detection result is obtained, the method further includes: If the detection result indicates that the integrated text is inconsistent with the facts, it is determined that the pre-trained language model does not have the ability to answer the corresponding factual knowledge questions in the factual knowledge question assessment set.

7. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the method for assessing the ability to answer factual knowledge questions as claimed in any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for evaluating the ability to answer factual knowledge questions according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for evaluating the ability to answer factual knowledge questions according to any one of claims 1 to 6 are implemented.

Images