Evaluation method, device and storage medium of multi-hop question answering model

Abstract

The embodiments of this specification provide a method, device and storage medium for evaluating a multi-hop question-answering model, the method comprising: obtaining a target question and an original text corresponding to a true answer to the target question in a database of the multi-hop question-answering model; extracting a reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to infer the answer to the target question; generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain; adding the at least one interference text to the original text to obtain a test text; and evaluating the anti-interference ability of the multi-hop question-answering model according to the reasoning answer inferred from the test text by the multi-hop question-answering model, thereby achieving a targeted evaluation of the model and accurately finding the weaknesses of the model.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and more particularly to an evaluation method, device, and storage medium for a multi-hop question-answering model. Background Art

[0002] Question answering (QA) is an important research field in natural language processing (NLP). In this field, researchers aim to build a system that can automatically answer questions asked by humans in "natural language form". The research results in the field of question answering have long been popularized in the lives of each of us. For example, when you get up in the morning and ask your smart voice assistant "What's the weather like today?", you will get an answer like this: "Today is sunny, and the temperature is 15-22 degrees Celsius."

[0003] Single-hop question answering refers to obtaining an answer by simply matching a question with a single context. To verify the accuracy of a single-hop question answering model, it is usually necessary to evaluate the single-hop question answering model, such as modifying the entire question by replacing entities and adding new sentences to the input context to construct interference, so as to determine the stability of the model, that is, whether it can still accurately answer the question in the case of interference.

[0004] Multi-hop question answering is a very challenging task in the field of natural language processing and has attracted widespread attention. Compared with traditional single-hop question answering tasks, multi-hop question answering requires aggregating multiple facts from different contexts and performing compound reasoning on them to find the answer. Therefore, the evaluation method of single-hop question answering models is not suitable for multi-hop question answering models because entities play a key role in connecting different contexts to study the reasoning chain. Replacing entities may make the interfering text completely irrelevant to the question, resulting in limited distracting effect. In addition, this practice cannot make the answer prediction traceable and cannot determine at which link the model fails.

[0005] Existing evaluations of multi-hop question answering models measure standard metrics for answer prediction, but are insufficient or undirected for testing whether the model actually performs multi-hop reasoning. Summary of the invention

[0006] The purpose of the embodiments of this specification is to provide a method, device and storage medium for evaluating a multi-hop question-answering model, so as to evaluate the model in a targeted manner and accurately find the weaknesses of the model.

[0007] To solve the above problems, an embodiment of the present specification provides an evaluation method for a multi-hop question-answering model, the method comprising: obtaining a target question and an original text corresponding to the true answer to the target question in a database of the multi-hop question-answering model; extracting an inference chain in the original text; the inference chain represents the relationship between multiple entities used to infer the answer to the target question; generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the inference chain; adding the at least one interference text to the original text to obtain a test text; and evaluating the anti-interference ability of the multi-hop question-answering model according to the inference answer inferred from the test text by the multi-hop question-answering model.

[0008] Furthermore, the extracting of the reasoning chain in the original text includes: constructing an entity graph based on the original text; the entity graph includes multiple triples, each triplet represents two entities and the relationship between the two entities; extracting the question entity in the target question; matching the real answer entity corresponding to the question entity from the entity graph; extracting the shortest path from the question entity to the real answer entity in the entity graph as the reasoning chain.

[0009] Furthermore, constructing an entity graph based on the original text includes: calculating the relevance between the target question and each paragraph in the original text based on a pre-trained BERT model; and selecting target paragraphs whose relevance is greater than a preset threshold to construct an entity graph.

[0010] Furthermore, the type of the inference chain is determined in the following manner: a preset model is used to extract features of the inference chain; and the type of the inference chain is determined according to the features of the inference chain.

[0011] Furthermore, generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain includes: when the type of the reasoning chain is a bridging type, modifying the attribute relationship of the question entity in the target question to generate a false answer; wherein the relationships between the multiple entities in the bridging type reasoning chain are crossed; obtaining the number of relationships of the jumping point in the reasoning chain; the jumping point represents an entity associated with multiple relationships; and constructing interference texts of the number of relationships according to the false answer.

[0012] Furthermore, generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain includes: when the type of the reasoning chain is comparative, modifying the attribute relationship of the question entity in the target question to generate a false answer; wherein the relationship between the multiple entities in the comparative reasoning chain does not intersect; and constructing the interference text according to the false answer.

[0013] Furthermore, the evaluating the anti-interference ability of the multi-hop question-answering model based on the inference answer inferred from the test text by the multi-hop question-answering model includes: comparing the inference answer and the true answer; when the inference answer and the true answer are the same, determining that the anti-interference ability of the multi-hop question-answering model is strong; when the inference answer and the true answer are different, determining that the anti-interference ability of the multi-hop question-answering model is weak.

[0014] To solve the above problems, an embodiment of the present specification also provides an evaluation device for a multi-hop question-answering model, the device comprising: an acquisition module, used to acquire the target question and the original text corresponding to the true answer to the target question in the database of the multi-hop question-answering model; an extraction module, used to extract the reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to reason to obtain the answer to the target question; a generation module, used to generate at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain; a testing module, used to add the at least one interference text to the original text to obtain a test text; and an inference module, used to evaluate the anti-interference ability of the multi-hop question-answering model according to the inference answer inferred from the test text by the multi-hop question-answering model.

[0015] To solve the above problems, an embodiment of the present specification also provides an electronic device, including: a memory for storing a computer program; a processor for executing the computer program to achieve: obtaining the original text corresponding to the true answer to the target question in the database of the target question and the multi-hop question and answer model; extracting the reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to infer the answer to the target question; generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain; adding the at least one interference text to the original text to obtain a test text; and evaluating the anti-interference ability of the multi-hop question and answer model according to the reasoning answer inferred from the test text by the multi-hop question and answer model.

[0016] To solve the above problems, an embodiment of the present specification also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, implement: obtaining the original text corresponding to the true answer to the target question in the database of the target question and the multi-hop question-answering model; extracting the reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to infer the answer to the target question; generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain; adding the at least one interference text to the original text to obtain a test text; and evaluating the anti-interference ability of the multi-hop question-answering model according to the reasoning answer inferred from the test text by the multi-hop question-answering model.

[0017] It can be seen from the technical solutions provided in the above embodiments of this specification that, in the embodiments of this specification, the original text corresponding to the real answer to the target question in the database of the target question and the multi-hop question-answering model can be obtained; the reasoning chain in the original text is extracted; the reasoning chain represents the relationship between multiple entities used to infer the answer to the target question; at least one interference text that interferes with the relationship between the multiple entities is generated according to the type of the reasoning chain; the at least one interference text is added to the original text to obtain a test text; the anti-interference ability of the multi-hop question-answering model is evaluated according to the reasoning answer inferred from the test text by the multi-hop question-answering model. The method provided in the embodiments of this specification classifies the reasoning chain from question to answer into different reasoning types, and reversely modifies the corresponding part of the question to generate interference text, thereby realizing a directional evaluation of the model and accurately finding the weaknesses of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is a flow chart of a method for evaluating a multi-hop question-answering model according to an embodiment of this specification;

[0020] Figure 2 A schematic diagram of the reasoning chain of the original text corresponding to the target question 1 of the embodiment of this specification;

[0021] Figure 3 A schematic diagram of the reasoning chain of the original text corresponding to the target question 2 of the embodiment of this specification;

[0022] Figure 4 A schematic diagram of the reasoning chain of the original text corresponding to the target question 3 of the embodiment of this specification;

[0023] Figure 5 A schematic diagram of the reasoning chain of the original text corresponding to the target question 4 of the embodiment of this specification;

[0024] Figure 6 A schematic diagram of a reasoning chain including interference text corresponding to the target problem 1 of the embodiment of this specification;

[0025] Figure 7 A schematic diagram of a reasoning chain including interference text corresponding to the target problem 2 of the embodiment of this specification;

[0026] Figure 8 A schematic diagram of a reasoning chain including interference text corresponding to the target question 3 of the embodiment of this specification;

[0027] Fig. 9 A schematic diagram of a reasoning chain including interference text corresponding to the target question 4 of the embodiment of this specification;

[0028] Fig.10 This is a functional structure diagram of an electronic device according to an embodiment of this specification;

[0029] Fig.11 This is a functional structure diagram of an evaluation device for a multi-hop question-answering model according to an embodiment of this specification. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0031] Unlike traditional information retrieval and database retrieval, the field of question answering aims to study questions given in "natural language form" rather than structured query language, which is more in line with application scenarios in daily life. However, the ambiguity of natural language also makes it very difficult to accurately understand the problem. At the same time, the data sources of question answering may be diverse. In different scenarios, structured knowledge graphs and unstructured texts may be potential data sources. Therefore, studying corresponding reasoning algorithms for different data sources is also one of the current hot directions in the field of question answering.

[0032] The knowledge graph is a data structure that stores human knowledge in the form of a "multi-relation directed graph". Each node in the knowledge graph represents an entity, and the directed edge between two nodes represents the relationship between them. For example, "Yao Ming" is an entity, "Shanghai" is also an entity, and the relationship between them is "born in". Such a triple "Yao Ming, born in, Shanghai" represents a fact. As a data source for question and answer, compared with unstructured text data, structured knowledge graphs represent human knowledge in a clearer and more accurate way, thus bringing unprecedented development opportunities for the construction of high-quality question and answer systems.

[0033] Question Answering over Knowledge Graphs refers to a question-answering scenario that uses knowledge graphs as the main data source. For a given question, we reason based on the knowledge graph to get the answer. This technology has been widely used in the industry in related intelligent search and recommendation businesses, the most famous of which is Google's search engine business. For example, for such a simple question, "Where was Yao Ming born?", the answer obtained through the triple "Yao Ming, born in, Shanghai" in the knowledge graph is the entity "Shanghai".

[0034] Simply put, single-hop question answering refers to questions that can be answered using a single triple. For example, for the above question "Where is Yao Ming born?", you can use the triple "Yao Ming, born in, Shanghai" to get the answer "Shanghai". Multi-hop question answering refers to questions that cannot be answered using a single triple. For example, for the question "Who are the directors of the movies starring Jackie Chan?", this question requires a multi-hop reasoning path formed by multiple triples to be answered. For example: through these triples "Jackie Chan, starring, New Police Story", "New Police Story, director, Chen Musheng", we infer that "Chen Musheng" is a correct answer. Of course, since Jackie Chan has starred in many movies, this is just one of the answers. Due to space reasons, they are not listed one by one in this manual.

[0035] To verify the accuracy of a single-hop question answering model, it is usually necessary to evaluate the single-hop question answering model, for example, by modifying the entire question by replacing entities and adding new sentences to the input context to build interference, so as to determine the stability of the model, that is, whether it can still accurately answer the question under interference. The evaluation method of the single-hop question answering model is not applicable to the multi-hop question answering model because entities play a key role in connecting different contexts to study the reasoning chain. Replacing entities may make the interference text completely irrelevant to the question, resulting in limited distraction effect. In addition, this approach cannot make the answer prediction traceable or determine at which link the model fails. Existing evaluations of multi-hop question answering models are just standard metrics to measure answer predictions, but they are insufficient or undirected to evaluate whether the model actually performs multi-hop reasoning.

[0036] Considering that if it is possible to interfere with unused jumps by changing the relationship of the reasoning chain (triples) in the reasoning process of multi-hop question answering according to the type of question answering, it is expected to solve the problem that the evaluation of the multi-hop question answering model in the prior art cannot accurately locate the weaknesses of the model. Based on this, the embodiments of this specification provide a method, device and storage medium for evaluating a multi-hop question answering model to achieve targeted evaluation of the model and accurately find the weaknesses of the model.

[0037] See also Figure 1 . An embodiment of the present specification provides an evaluation method for a multi-hop question-answering model. In an embodiment of the present specification, the subject that executes the evaluation method for the multi-hop question-answering model may be an electronic device with a logical operation function, and the electronic device may be a server. The server may be an electronic device with a certain computing and processing capability. It may have a network communication unit, a processor, a memory, and the like. Of course, the server is not limited to the above-mentioned electronic device with a certain entity, and it may also be software running in the above-mentioned electronic device. The server may also be a distributed server, and may be a system with multiple processors, memories, network communication modules, etc. operating in collaboration. Alternatively, the server may also be a server cluster formed by several servers. The method may include the following steps.

[0038] S110: Obtain the target question and the original text corresponding to the true answer to the target question in the database of the multi-hop question-answering model.

[0039] In some embodiments, the database of the multi-hop question-answering model can be a HotpotQA dataset, which includes multiple sets of question-answer pairs, i.e., combinations of questions and answers, and texts related to the extracted answers, which are the original texts corresponding to the true answers to the target questions mentioned above.

[0040] For example, for the target question "Which island is located 808 nautical miles northwest of the capital of Hawaii?" the original text corresponding to this question in the database can be "Laysan, located 808 nautical miles northwest of Honolulu, is one of the northwestern Hawaiian Islands. It includes a 1,016-acre land area of ​​approximately 1 acre. It is an atoll island, although the land completely surrounds a shallow central lake about 8 feet above sea level, with a salinity of about three times that of the ocean." and "Honolulu is the capital and largest city of the U.S. state of Hawaii. It is an unincorporated part and county seat of the City and County of Honolulu on the island of Oahu. The city is the main gateway to Hawaii and the main gateway to the United States."

[0041] S120: Extracting reasoning chains in the original text; the reasoning chains represent the relationship between multiple entities used to infer the answer to the target question.

[0042] In multi-hop question answering, questions need to be answered using multi-step reasoning. In each step, multiple entities are involved, and there are some relationships between these entities. The reasoning chain represents the multiple entities involved and the relationships between these entities in the multi-step reasoning process from question to answer.

[0043] In some embodiments, extracting the reasoning chain in the original text includes: constructing an entity graph based on the original text; the entity graph includes multiple triples, each triple representing two entities and the relationship between the two entities; extracting the question entity in the target question; matching the real answer entity corresponding to the question entity from the entity graph; extracting the shortest path from the question entity to the real answer entity in the entity graph as the reasoning chain.

[0044] Specifically, the entity graph includes nodes and edges, wherein the nodes represent entities, and the edges represent the relationship between entities. In an embodiment of the present specification, the entity graph is composed of a plurality of triples, each of which may include two nodes and an edge connecting the two nodes, that is, each triple represents two entities and the relationship between the two entities.

[0045] For example, for target question 1, “In what year was the singer who sang “Orange” born?”

[0046] The original text includes:

[0047] First dance: "Orange" is a song sung by Irish theatre actor David O'Doherty;

[0048] Second jump: David Nicholas O'Doherty (born December 18, 1975 in Dublin, Ireland) is an Irish comedian, writer, musician, actor and playwright.

[0049] Real answer: 1975.

[0050] The reasoning chain for the original text corresponding to target question 1 is as follows Figure 2 As shown, it contains three entities and two triplets.

[0051] For objective question 2, “Who is the grandson of Aurelia Plath and is a fisheries biologist?”

[0052] The original text includes:

[0053] First jump: Nicholas Farrar Hughes is a fisheries biologist;

[0054] Second jump: Nicholas Farrar Hughes is the grandson of Aurelia Plath.

[0055] Real answer: Nicholas Farrar Hughes.

[0056] The reasoning chain of the original text corresponding to target question 2 is as follows Figure 3 As shown, it contains three entities and two triplets.

[0057] For target question 3, "Who was born earlier, Emma Bull or Virginia Woolf?"

[0058] The original text includes:

[0059] First jump: Emma Bull (born December 13, 1954) is an American science fiction and fantasy writer;

[0060] Virginia Woolf (25 January 1882 – 28 March 1941) was a British writer, considered one of the most important modernists of the twentieth century.

[0061] Real answer: Virginia Woolf.

[0062] The reasoning chain of the original text corresponding to target question 3 is as follows Figure 4 As shown, it contains four entities and two triplets.

[0063] For target question 4 “Are Thomas H. Ince and Joseph McGrath the same nationality?”

[0064] The original text includes:

[0065] First Jump: Thomas H. Ince (November 16, 1880 - November 19, 1924) was an American silent film producer, director, screenwriter, and actor.

[0066] Second jump: Joseph McGrath (born in Glasgow in 1930) is a Scottish film and television director and screenwriter.

[0067] Real answer: No.

[0068] The reasoning chain of the original text corresponding to target question 4 is as follows Figure 5 As shown, it contains four entities and two triplets.

[0069] In some embodiments, in order to construct the original text as Figure 2-Figure 5The entity graph shown in the figure first extracts the entities associated with the target question, and then extracts the relations between these entities, thereby completing the construction of the entity graph. Specifically, for each embodiment, an entity graph can be attempted to be constructed on all selected paragraphs. First, some existing tools can be used, such as using the Stanford Corenlp toolkit (Manning et al. 2014) to identify named entities (respectively denoted as Eq and Ec) from the question and input context. At the same time, the OpenIE5 toolkit can also be used to extract entity parameter-relationship-entity parameter triples for each sentence. The extracted relations can be noun phrases or verb phrases, corresponding to attribute relations and action relations, respectively. If the entity parameters and relations are too long, they can be re-extracted to obtain more implicit triples and clean entities. If applicable, the entity parameters will be further modified to align with the named entities in the pre-extraction set (Eq and Ec). If the entity parameter is a pronoun, it is directly replaced with the subject entity of the previous sentence.

[0070] In cross-document QA tasks, the input context may include many irrelevant paragraphs, which introduce a large number of entities and triples, making the entity graph too large to perform effective reasoning chain search. Therefore, a paragraph filtering module can be used to discard irrelevant paragraphs. In some embodiments, constructing an entity graph based on the original text includes: calculating the relevance of the target question to each paragraph in the original text based on a pre-trained BERT model; and selecting target paragraphs whose relevance is greater than a preset threshold to construct an entity graph.

[0071] Specifically, BERT stands for Bidirectional Encoder Representation from Transformers, which is a pre-trained language representation model. It emphasizes that it no longer uses the traditional unidirectional language model or the shallow splicing of two unidirectional language models for pre-training as in the past, but adopts a new masked language model (MLM) to generate deep bidirectional language representation. In the embodiment of this specification, the target question and each paragraph in the original text can be input into the pre-trained BERT model, and a probability score is output through the classification layer to represent the relevance, or more specifically, whether there are supporting facts in the paragraph. Paragraphs with relevance scores higher than the predefined threshold are selected for subsequent entity graph construction.

[0072] S130: Generate at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain.

[0073] The reasoning chain is formed by taking the shortest path from each question entity to the answer entity (if applicable) along the constructed entity graph. The chain actually encodes the reasoning mechanism and can be used to design specific interference strategies. Based on the characteristics of the reasoning chain, the reasoning chains in multi-hop question answering can be divided into the following types:

[0074] Bridging type: requires sequential reasoning. We must first infer the bridge entity and then use it to find the second-hop answer. Projected onto the graph, the reasoning chain is unidirectional, with one or more bridge nodes connecting the question entity and the answer node. The reasoning process is to match the relationship edges step by step.

[0075] Crossover type: requires the answer to meet multiple conditions at the same time. The reasoning chain involves at least two independent paths, in which different question entity nodes independently point to the answer. Any entity with a disconnected path is not the answer.

[0076] Comparison type: compares the attributes of two entities. These questions usually do not have a general connected path like the first two types of questions. The two problem entities have parallel edges pointing to their respective attribute nodes, and subsequent operations will be performed on these two attribute nodes to obtain the final answer, which is usually one of the two problem entities, and sometimes their common attributes.

[0077] Yes / No type: asks whether two entities have the same attributes, the answer is "yes" or "no". Similar to the comparison type, a special operation node "is same?" is introduced to form a connected reasoning chain.

[0078] In some embodiments, the type of the inference chain may be determined in the following manner: extracting features of the inference chain using a preset model; and determining the type of the inference chain according to the features of the inference chain.

[0079] Specifically, in the HotpotQA dataset, both bridge and cross reasoning chains are classified as bridge, and both comparison and yes / no are classified as comparison. With these existing labels, the classification of comparison and yes / no is very simple, just checking whether the comparison answer is "yes" or "no".

[0080] For both bridge and cross types, a model based on reasoning chains is designed to predict reasoning types and question spans with different hop counts. Since there may be multiple entities in a question, the entity with the longest shortest path length to the answer node can be selected. The question entity node has the most hops to the answer, so it is most likely to be the starting point of reasoning and contain the most information. If the chain contains more than two hops, the second hop and the subsequent hops are combined as the second hop. Formally, the chain is defined as [HOP1]ent q rel1[HOP2]ent b1rel2ent b2 rel3…, where [HOP1] and [HOP2] are two special tokens. The model takes the concatenation of the question word and the chain word as input sequence S = [CLS] question [SEP] chain [SEP]. The input is encoded using the pre-trained BERT model:

[0081]

[0082] Where n and h represent the maximum input length and the number of BERT hidden states, respectively, and U represents the encoded output.

[0083] Then apply a question mask M∈R n×h With the trainable parameter matrix W1∈R h×2 , use the softmax function to calculate the word logits, that is, the probability of each word being the starting position and the ending position of the first jump subproblem:

[0084]

[0085] By maximizing the joint probability ind start ,ind end To infer the starting position P start and end position P end :

[0086]

[0087] Among them, L q Indicates the length of the problem, i and j are in different positions.

[0088] The model is optimized based on the joint cross entropy loss function:

[0089]

[0090] in, represents the joint cross entropy loss function, represents the loss function at the starting position, Represents the loss function of the end position, λ is a variable parameter, Represents the loss function of the category.

[0091] To generate interference text, the range of question text corresponding to the selected target jump can first be semantically modified to ask other questions. In some embodiments, for a cross-type reasoning chain, a jump point can be randomly selected for modification, because the first jump and the second jump are equivalent in the reasoning chain; for a bridge-type reasoning chain, the relationship of the first jump can be modified, because it is more likely to bring successful interference when the interference reasoning starts at the first jump. Although many adversarial interference works attempt to increase interference by changing entities, since entities are important in multi-hop question answering, modifying entities may make the interference text irrelevant to the original question and original text, so interfering with relationships is a better solution. Generally, relationships can be attribute relationships based on nouns (e.g., the first son) or action relationships based on verbs (e.g., born). Therefore, the modified objects can be common nouns, adjectives, verbs, etc. (except auxiliary verbs and model verbs), while named entities (including people, organizations, locations, and numbers) remain unchanged. First, you can try to replace them with antonyms in WordNet (WordNet is an English dictionary based on cognitive linguistics designed by psychologists, linguists, and computer engineers at Princeton University. It does not just arrange words in alphabetical order, but also forms a "network of words" according to the meaning of the words.) If it fails, change them to the nearest word in the word vector space. In addition, you can restrict the replacement words to have the same NER type and part of speech, and not share the same stem with the original word. In the above example, "first son" will be changed to "last daughter", and "born in" will be changed to "named in".

[0092] In some embodiments, generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain includes: when the type of the reasoning chain is a bridging type, modifying the attribute relationship of the question entity in the target question to generate a false answer; wherein the relationships between the multiple entities in the bridging type reasoning chain are crossed; obtaining the number of relationships of the jumping point in the reasoning chain; the jumping point represents an entity associated with multiple relationships; and constructing interference texts of the number of relationships according to the false answer.

[0093] The new relation generated by the above steps cannot be assigned to the original answer because it may contradict the original context. Therefore, a fake answer can be generated to ensure compatibility. For example, named entities are extracted from all answers in the training dataset and classified into different NER types to build a fake answer set. For each non-stop word to be attacked in the answer text, first try to find an alternative word using the same method as mentioned above. If it fails, randomly select an answer with the same type from the fake answer set. For example, the alternative word for "1998" is "1999", and the alternative word for "Nicholas Farrar Hughes" is "Otto Emil Plath" in the fake answer set.

[0094] Finally, the modified question and the fake answer can be combined to generate the final interference text. For example, for the bridging type of reasoning chain, the corresponding question is question 1 above, "In which year was the singer who sang "Orange" born?"

[0095] First sub-question: Who is the singer who sang "Orange"?

[0096] Second sub-question: In which year was the “bridge entity” created?

[0097] Real answer: 1975.

[0098] False answer: 1950.

[0099] First jump interference text: The producer of "Orange" was born in 1950;

[0100] Second jump interference text: The singer who sang "Orange" was named in 1950.

[0101] For an inference chain containing interfering text, Figure 6 shown.

[0102] For another example, for a cross-type reasoning chain, the corresponding question is Question 2 above: “Who is the grandson of Aurelia Plath and a fishery biologist?”

[0103] The first sub-question: Who is Aurelia Plath’s grandson?

[0104] Second sub-question: Who is a fishery biologist?

[0105] Real answer: Nicholas Farrar Hughes.

[0106] False answer: Otto Emil Plath.

[0107] First Jump Interference Text: Otto Emil Plath is Aurelia Plath's nephew and a fishery biologist;

[0108] Second jump interference text: Otto Emil Plath is the grandson of Aurelia Plath and is a forestry geologist.

[0109] For an inference chain containing interfering text, Figure 7 As shown in Figure 2, a distractor text “Otto Emil Plath is the nephew of Aurelia Plath and is a fishery biologist” is constructed for target question 2. The generated sentence is very similar to target question 2, with one sub-question completely overlapping and the other sub-question slightly modified while the syntactic structure remains unchanged. Obviously, “Otto Emil Plath” is not a valid answer because it only satisfies part of the original target question. The generated distractors are inserted independently into random positions of each paragraph, and the supporting fact labels are also modified if necessary.

[0110] In some embodiments, generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain includes: when the type of the reasoning chain is comparative, modifying the attribute relationship of the question entity in the target question to generate a false answer; wherein the relationship between the multiple entities in the comparative reasoning chain does not intersect; and constructing the interference text according to the false answer.

[0111] Specifically, the above-mentioned comparison-type reasoning chains and yes / no-type reasoning chains can both be classified as comparisons. The relationships between multiple entities in these two types of reasoning chains do not cross, and different jumps are equivalent in the reasoning chain. Considering that there is a special comparison operation node in the reasoning chain, a concept path that directly performs the same operation can be added. First, a syntactic component parse tree is constructed, and the two comparators are identified by extracting the two syntactic units before and after the word "and" or "or". In order to obtain more likely interference, in terms of setting false answers: (1) for the comparison type, the question entity other than the real answer can be set as a false answer; (2) for the yes / no type, a negative sentence can be constructed for the original answer "yes", otherwise an affirmative sentence can be constructed. The last step is to convert the modified question and false answer into a syntactically correct declarative sentence.

[0112] For example, for a comparative type of reasoning chain, the corresponding question is Question 3 above: "Who was born earlier, Emma Bull or Virginia Woolf?"

[0113] Real answer: Virginia Woolf.

[0114] Fake answer: Emma Bull.

[0115] Distracting text: Emma Bull is named earlier than Virginia Woolf.

[0116] For an inference chain containing interfering text, Figure 8 shown.

[0117] For a yes / no type of reasoning chain, the corresponding question is Question 4 above, "Are Thomas H. Ince and Joseph McGrath of the same nationality?"

[0118] Real answer: No.

[0119] False answer: Yes.

[0120] Distracting Text: Thomas H. Ince and Joseph McGrath have the same identity.

[0121] For an inference chain containing interfering text, Fig. 9 shown.

[0122] S140: Add the at least one interference text to the original text to obtain a test text.

[0123] In some embodiments, after at least one interference text that interferes with the relationship between the multiple entities is generated according to the type of the reasoning chain, the interference text may be inserted into the original text to obtain a test text.

[0124] S150: Evaluate the anti-interference capability of the multi-hop question answering model according to the inference answer inferred by the multi-hop question answering model from the test text.

[0125] In some embodiments, the target question corresponding to the test text can be input into the multi-hop question-answering model to be tested. The multi-hop question-answering model can perform inference from the test text to obtain an inference answer, and the anti-interference ability of the multi-hop question-answering model can be determined based on the inference answer.

[0126] In some embodiments, evaluating the anti-interference ability of the multi-hop question-answering model based on the inference answer inferred from the test text by the multi-hop question-answering model includes: comparing the inference answer and the true answer; when the inference answer and the true answer are the same, determining that the anti-interference ability of the multi-hop question-answering model is strong; when the inference answer and the true answer are different, determining that the anti-interference ability of the multi-hop question-answering model is weak.

[0127] Specifically, by comparing the inference answer and the true answer, if the inference answer and the true answer are the same, it means that the multi-hop question answering model has passed the test and has a strong anti-interference ability; if the inference answer and the true answer are different, it means that the multi-hop question answering model has passed the test and has a weak anti-interference ability.

[0128] In some embodiments, the weaknesses of the multi-hop question-answering model can also be determined based on the reasoning answers. Specifically, the reasoning answers can be used to determine which link in the reasoning chain of the multi-hop question-answering model has a problem during the reasoning process, and then the multi-hop question-answering model can be retrained based on the link with the problem to make up for the weaknesses of the model.

[0129] The method provided in the embodiment of this specification can obtain the original text corresponding to the real answer of the target question in the database of the target question and the multi-hop question-answering model; extract the reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to infer the answer to the target question; generate at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain; add the at least one interference text to the original text to obtain a test text; evaluate the anti-interference ability of the multi-hop question-answering model according to the reasoning answer inferred from the test text by the multi-hop question-answering model. The method provided in the embodiment of this specification classifies the reasoning chain from question to answer into different reasoning types, and reversely modifies the corresponding part of the question to generate interference text, so as to achieve directional evaluation of the model and accurately find the weaknesses of the model.

[0130] The method embodiments provided in the above embodiments of this specification may be executed in a mobile terminal, a computer terminal or similar electronic devices. Fig.10 This is a functional structure diagram of an electronic device according to an embodiment of the present specification. The electronic device may include a memory and a processor.

[0131] In some embodiments, the memory can be used to store the computer program and / or module, and the processor implements various functions of the evaluation method of the multi-hop question-answering model by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the user terminal. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0132] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor may execute the computer instructions to implement the following steps: obtaining the original text corresponding to the true answer to the target question in the database of the target question and the multi-hop question-answering model; extracting the reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to infer the answer to the target question; generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain; adding the at least one interference text to the original text to obtain a test text; and evaluating the anti-interference ability of the multi-hop question-answering model according to the reasoning answer inferred from the test text by the multi-hop question-answering model.

[0133] In the embodiments of this specification, the functions and effects specifically implemented by the electronic device can be explained in comparison with other embodiments and will not be repeated here.

[0134] Fig.11 This is a functional structural diagram of an evaluation device for a multi-hop question-answering model according to an embodiment of the present specification. The device may specifically include the following structural modules.

[0135] An acquisition module 1110 is used to acquire the target question and the original text corresponding to the true answer to the target question in the database of the multi-hop question-answering model;

[0136] An extraction module 1120 is used to extract the reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to infer the answer to the target question;

[0137] A generating module 1130, configured to generate at least one interference text that interferes with the relationship between the plurality of entities according to the type of the reasoning chain;

[0138] A testing module 1140, configured to add the at least one interference text to the original text to obtain a test text;

[0139] The reasoning module 1150 is used to evaluate the anti-interference ability of the multi-hop question-answering model according to the reasoning answer inferred by the multi-hop question-answering model from the test text.

[0140] An embodiment of the present specification also provides a computer-readable storage medium for a method for evaluating a multi-hop question-answering model, wherein the computer-readable storage medium stores computer program instructions, which, when executed, implement the following: obtaining the target question and the original text corresponding to the true answer to the target question in the database of the multi-hop question-answering model; extracting the reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to reason to obtain the answer to the target question; generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain; adding the at least one interference text to the original text to obtain a test text; and evaluating the anti-interference ability of the multi-hop question-answering model according to the reasoning answer inferred from the test text by the multi-hop question-answering model.

[0141] In the embodiments of this specification, the above-mentioned storage medium includes but is not limited to random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), cache (Cache), hard disk (Hard Disk Drive, HDD) or memory card (Memory Card). The memory can be used to store the computer program and / or module, and the memory can mainly include a storage program area and a storage data area, wherein the storage program area can store an operating system, at least one application required for a function, etc.; the storage data area can store data created according to the use of the user terminal, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory. In the embodiments of this specification, the functions and effects specifically implemented by the program instructions stored in the computer-readable storage medium can be explained in comparison with other implementations, and will not be repeated here.

[0142] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment and the equipment embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0143] After reading this specification, those skilled in the art can conceive of any combination of some or all of the embodiments listed in this specification without creative work, and these combinations are also within the scope of disclosure and protection of this specification.

[0144] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0145] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0146] This specification can be used in many general or special computer system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0147] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0148] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A method for evaluating a multi-hop question answering model, characterized in that: The method comprises: Obtaining a target question and an original text corresponding to a true answer to the target question in a database of a multi-hop question-answering model; Extracting a reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to infer the answer to the target question; generating at least one interference text that interferes with the relationship between the plurality of entities according to the type of the reasoning chain; Adding the at least one interference text to the original text to obtain a test text; The anti-interference ability of the multi-hop question answering model is evaluated according to the inference answer obtained by the multi-hop question answering model from the test text.

2. The method according to claim 1, characterized in that The extracting of the reasoning chain in the original text comprises: Construct an entity graph according to the original text; the entity graph includes a plurality of triples, each triple representing two entities and the relationship between the two entities; Extracting question entities in the target question; Obtaining a true answer entity corresponding to the question entity by matching from the entity graph; The shortest path from the question entity to the true answer entity in the entity graph is extracted as the reasoning chain.

3. The method according to claim 2, characterized in that The constructing an entity graph according to the original text comprises: Calculate the relevance of the target question to each paragraph in the original text according to the pre-trained BERT model; Select target paragraphs whose relevance is greater than a preset threshold to build an entity graph.

4. The method according to claim 1, characterized in that The type of the chain of reasoning is determined as follows: Extracting features of the reasoning chain using a preset model; The type of the reasoning chain is determined according to the characteristics of the reasoning chain.

5. The method according to claim 1, characterized in that Generating at least one interference text that interferes with the relationship between the plurality of entities according to the type of the reasoning chain comprises: In the case where the type of the reasoning chain is a bridging type, modifying the attribute relationship of the question entity in the target question to generate a false answer; wherein the relationships between multiple entities in the bridging type reasoning chain are crossed; Acquire the number of relations of the jumping point in the reasoning chain; the jumping point represents an entity associated with multiple relations; Construct the relational number of interference texts according to the false answers.

6. The method according to claim 1, characterized in that Generating at least one interference text that interferes with the relationship between the plurality of entities according to the type of the reasoning chain comprises: In the case where the type of the reasoning chain is comparative, modifying the attribute relationship of the question entity in the target question to generate a false answer; wherein the relationship between multiple entities in the comparative reasoning chain does not cross; A distraction text is constructed based on the false answers.

7. The method according to claim 1, characterized in that The step of evaluating the anti-interference capability of the multi-hop question answering model according to the inference answer inferred from the test text by the multi-hop question answering model comprises: comparing the inferred answer with the true answer; When the inference answer and the true answer are the same, determining that the multi-hop question-answering model has a strong anti-interference ability; When the inferred answer and the true answer are different, it is determined that the anti-interference ability of the multi-hop question-answering model is weak.

8. An evaluation device for a multi-hop question-answering model, characterized in that: The device comprises: An acquisition module, used to acquire the target question and the original text corresponding to the true answer to the target question in the database of the multi-hop question-answering model; An extraction module, configured to extract a reasoning chain in the original text; the reasoning chain represents a relationship between multiple entities used to infer an answer to the target question; A generating module, configured to generate at least one interference text that interferes with the relationship between the plurality of entities according to the type of the reasoning chain; A testing module, used for adding the at least one interference text into the original text to obtain a test text; The inference module is used to evaluate the anti-interference ability of the multi-hop question-answering model according to the inference answer inferred by the multi-hop question-answering model from the test text.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement: obtaining a target question and an original text corresponding to a true answer to the target question in a database of a multi-hop question-answering model; extracting an inference chain in the original text; the inference chain represents a relationship between multiple entities used to infer the answer to the target question; generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the inference chain; adding the at least one interference text to the original text to obtain a test text; and evaluating the anti-interference capability of the multi-hop question-answering model according to an inference answer inferred from the test text by the multi-hop question-answering model.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed, the following are achieved: obtaining the original text corresponding to the real answer to the target question in the database of the target question and the multi-hop question-answering model; extracting the reasoning chain in the original text; the reasoning chain represents the relationship between multiple entities used to infer the answer to the target question; generating at least one interference text that interferes with the relationship between the multiple entities according to the type of the reasoning chain; adding the at least one interference text to the original text to obtain a test text; The anti-interference ability of the multi-hop question answering model is evaluated according to the inference answer obtained by the multi-hop question answering model from the test text.

Images