A text multi-hop reasoning question answering method using two-stage retrieval

Abstract

The present invention discloses a text multi-hop reasoning question answering method using two-stage retrieval, and the field of natural language processing technology, and the method includes two modules. First, in the context-aware knowledge retriever module, a dual encoder framework with a mask mechanism is used for pre-training, and then the representation of the learned hypothesis and knowledge is based on sentence-level dense retrieval to obtain candidate core knowledge. Secondly, in the entity-aware knowledge verification module, the accessibility of the hypothesis and knowledge is determined by the entity-level sparse matrix, which will be used to further verify whether there is a supporting relationship between the core knowledge and the hypothesis. This method combines the context-aware and entity-aware knowledge retrieval methods and applies them to the reasoning process, improves the blindness of entity-centric reasoning, and enables the model to simultaneously consider the contextual semantic matching and entity concept influence between the hypothesis and knowledge. In the field of scientific question answering that requires complex multi-hops, it has obvious performance advantages.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a text multi-hop reasoning question-answering method using two-stage retrieval. Background Art

[0002] Multi-hop reasoning question answering is a form of question that uses two or more pieces of evidence in a knowledge base to perform specific reasoning to arrive at an answer. In current research, whether it is based on graph-structured reasoning, methods that use entity concepts to identify reasoning paths, or reasoning based on text knowledge bases, it is necessary to pay significant attention to entity information in order to more accurately derive the content of the next retrieval. However, in these methods, the contextual semantics of the sentence is inevitably ignored, which increases the unreliability of the multi-hop reasoning process. Therefore, the special attention to entities limits the performance of the model to a certain extent, causing the model to introduce false evidence. In the iterative retrieval model, it will lead to continuous accumulation of errors, causing the model performance to deteriorate catastrophically. To address this problem, the present invention proposes a text multi-hop reasoning question answering method using two-stage retrieval, which considers the influence of sentence context semantics and entity concepts in stages, and solves the question answering task in complex reasoning scenarios. Summary of the Invention

[0003] This paper addresses the problem that existing methods overly focus on the role of entities, blinding the reasoning question-answering process. It proposes a multi-hop text reasoning question-answering method using two-stage retrieval. This method effectively leverages both sentence context and entity concepts, efficiently training models for multi-hop reasoning tasks.

[0004] In order to achieve the above technical objectives, the technical solution of the present invention is:

[0005] A text multi-hop reasoning question answering method using two-stage retrieval includes the following steps:

[0006] S1, question q and each answer option a={a1,a2,...,a M} to obtain the hypothesis set h={h1,h2,...,h M}, where M is the number of options in the question. The original multiple-choice question, which simply concatenates question q and answer option a using a flag, is converted into a textual entailment question.

[0007] S2. Divide the domain theoretical knowledge in the external text knowledge base into a core knowledge set E and a background knowledge set G, where the core knowledge e (e∈E) represents the core idea of the problem investigation, and the background knowledge g (g∈G) represents the auxiliary knowledge used to solve the problem;

[0008] S3. In the first stage context-aware knowledge retriever module, the hypothesis h corresponding to the correct answer is i Correct Hypothesis and will support the correct assumption The established core knowledge e is called correct core knowledge will obscure any correct assumptions about the concept of partial entities. And the corresponding correct core knowledge Input into the dual encoder model in the context-aware knowledge retriever module; combined with the idea of contrastive learning, the correct hypothesis of other problems in the same batch training The corresponding correct core knowledge Negatively correlated knowledge After training, the correct hypothesis is obtained With correct core knowledge The mapping relationship between them;

[0009] S4, input all core knowledge e into the knowledge encoder in the dual encoding model, and output the semantic representation vector of the core knowledge e; then assume h i Input into the question hypothesis encoder in the dual encoder model to obtain the hypothesis h i The semantic representation vector of the core knowledge is retrieved by performing cosine similarity retrieval with the representation vectors of all core knowledge e to obtain the support hypothesis h i Candidate core knowledge pool The candidate core knowledge pool corresponding to question q At the same time, the core knowledge in the candidate core knowledge pool is obtained With the assumption h i The contextual semantic matching score between them;

[0010] S5. In the second stage entity perception knowledge verification module, the reachability matrix between entities in the background knowledge set G is constructed, and the Kuhn-Munkres algorithm is used to verify the hypothesis h i Entities in the candidate knowledge pool and core knowledge in the candidate knowledge pool Calculate the reachability between entities and obtain the core knowledge e j and assuming h i Entity matching score between ;

[0011] S6. Perform a weighted sum of the scores obtained by the context-aware knowledge retriever module and the entity-aware knowledge verification module; for the i-th hypothesis h in the question q with M options i , the candidate core knowledge with the highest score Assuming h i The prediction core knowledge is used to predict the hypothesis with the highest score (confidence) as the final answer to the question, and the dual encoder model is fine-tuned to minimize the loss value through the optimizer.

[0012] A further improvement of the present method is that: the conversion of the multiple-choice question-and-answer question into a text-implied question in step S1 includes: treating the core knowledge as a premise and the combination of the question and answer options as a hypothesis; for interrogative sentence questions, using lexical, grammatical and dependency syntax analysis methods to form semantically coherent and grammatically correct question hypotheses; for fill-in-the-blank questions, identifying the blanks and filling them with answer options.

[0013] A further improvement of the present method is that: in the step S2, the core knowledge is a text with strong theoretical nature and abstract expression; the background knowledge includes a short description text including entity attributes, entity usage and entity synonyms.

[0014] A further improvement of this method is that step S3 includes the following steps:

[0015] S3.1. Use the SpaCy text processing tool to extract all nouns and noun phrases from the background knowledge as entities. Semi-automatically reduce the noise extracted by the tool after analyzing the entity occurrence frequency and entity meaning to form an entity vocabulary.

[0016] S3.2. Select the correct hypothesis based on the entity vocabulary Entities in the , and randomly mask a certain proportion of entities, will correctly assume Input the problem hypothesis encoder into the dual encoding model and simultaneously convert its corresponding correct core knowledge Input into the knowledge encoder in the dual encoder model, and calculate the distance between the two by cosine similarity;

[0017] S3.3. Correct Hypothesis for Other Problems in the Same Training Batch Corresponding core knowledge Negatively correlated knowledge The loss function is optimized as the negative log-likelihood value of the correct core knowledge. The loss function is as follows:

[0018]

[0019] The further improvement of this method is that the acquisition process of the core knowledge pool in step S4 is as follows: the acquisition process of the core knowledge pool in step S4 is as follows: for all hypotheses h of question q i All entities with the same position order are masked, and the cosine similarity is used to obtain the hypothesis h after each mask. i The top K core knowledge Right now Where L is the size of the knowledge base, sim(·) represents the function for calculating cosine similarity, top k{·} Take the K elements with the highest scores in the set; then for each hypothesis h i Candidate core knowledge Then sort and get the top K core knowledge corresponding to question q as the candidate knowledge pool for question q

[0020] The further improvement of this method is that the step S5 includes: first extracting entities from the background knowledge, and constructing the reachability matrix P between the entities through the BFS algorithm, and then extracting the core knowledge e from the candidate knowledge pool. j and assuming h i Entities in the calculation hypothesis h i Entities in the candidate knowledge pool and core knowledge e j The reachability score between entities is calculated as follows:

[0021]

[0022] Among them, I(c i ,c j ) represents the symbolic function of whether entity i can reach entity j, |path(i,j)| represents the length of the path passed by the two entities; T h is a set of special entities of hypothesis h, where special entities refer to entities that do not appear simultaneously in all hypotheses of the same problem, S(c j ,T h ) is a symbolic function of whether entity j belongs to a special entity set; to prevent division by 0, a very small hyperparameter ε is added; is the i-th entity in the core knowledge sentence, is the jth entity in the hypothesis sentence; after obtaining the reachability score between the entity set in the hypothesis sentence and the entity set in the core knowledge, the core knowledge e is obtained using the Kuhn-Munkres algorithm. j and assuming h i The score KM(h i ,e j ).

[0023] A further improvement of this method is that the loss function of the dual encoder model in step S6 consists of two parts, namely the cross entropy loss based on the final answer and the cross entropy loss based on the knowledge pool, as follows:

[0024] Loss = L local +L global

[0025]

[0026]

[0027] Where N represents the sample size, M represents the number of options; y={y1,y2…,y M} is the one-hot representation of the answer label; p i represents the probability that the i-th choice is the correct answer, Represents the average score of all core knowledge corresponding to the i-th hypothesis in the core knowledge pool.

[0028] Due to the adoption of the above-mentioned technical solution, the technical progress achieved by the present invention is as follows: the present invention considers both contextual semantics and entity concept information in the model, first divides the domain knowledge into core knowledge and background knowledge, and then learns the mapping relationship between the correct hypothesis and the core knowledge after masking the entity through a pre-training process. The masking mechanism is to prevent the entity from dominating the mapping process. At the same time, the entity-aware knowledge retrieval module can represent the hypothesis and core knowledge at the entity level to further verify the accuracy of the core knowledge. The present invention reduces the blindness of the multi-hop reasoning process that relies solely on entity concepts, and effectively combines the retrieval method based on sentence context and entity concepts. In scientific question-answering datasets that require complex reasoning, the accuracy rate is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 It is an operation flow chart of the present invention;

[0031] Figure 2 This is a context-semantic-aware knowledge retrieval model diagram of the present invention;

[0032] Figure 3 The entity-aware knowledge entity link graph of the present invention;

[0033] Figure 4 It is the overall model diagram of the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] Figure 1This is a flowchart of a specific implementation of the text multi-hop reasoning question answering method using two-stage retrieval in the present invention. The specific implementation process and working principle of the present invention are as follows:

[0036] (1) The present invention first solves the question q and the answer option a={a1, a2, ..., a M} to obtain the hypothesis set h={h1,h2,...,h M Specifically, for interrogative sentences, the algorithm analyzes properties such as part of speech and dependency relationships in the question sentence and answer options, and automatically combines the question and answer options into a semantically coherent declarative sentence using an algorithm. For underlined fill-in-the-blank questions, the algorithm directly replaces the underline in the question sentence with the answer options.

[0037] (2) Figure 2 As shown, in the context semantic perception knowledge retrieval model, the present invention proposes a dual encoding training method combined with a neural network masking mechanism, which masks all correct hypotheses of some entities. And the corresponding correct core knowledge Input into the dual encoder model in the context-aware knowledge retriever module; combined with the idea of contrastive learning, the correct hypothesis of other problems in the same batch training Corresponding core knowledge Negatively correlated knowledge The loss function is optimized as the negative log-likelihood value of the correct core knowledge. That is, the loss function is Where sim(·) represents the function for calculating cosine similarity, E H (·) is defined as the hypothesized dense representation generated by the encoder, E E (·) is the knowledge-dense representation generated by the encoder.

[0038] (3) After obtaining the context-semantic-aware knowledge retriever through pre-training, all the hypotheses h for question q i All entities with the same position order are masked, and the cosine similarity is used to obtain the hypothesis h after each mask. i The top K core knowledge Right now Where L is the size of the knowledge base, sim(·) represents the function for calculating cosine similarity, top k {·} Take the K elements with the highest scores in the set; then for each hypothesis h i Candidate core knowledge Then sort and get the top K core knowledge corresponding to question q as the candidate knowledge pool for question q Where M is the number of options in the question.

[0039] (4) Figure 3As shown in the figure, in the entity-aware knowledge verification model, the present invention only considers that entities in short background knowledge texts containing "kind of," "part of," "example of," etc. have reachable relationships. Therefore, by connecting the same entity nodes in each common sense text, the BSF algorithm can be used to determine whether there is a reachable path between two arbitrary entities, ultimately obtaining the entity link relationship graph P.

[0040] (5) When there are many reasoning steps, even if there is no common entity concept between the hypothesis sentence and the core knowledge, we believe that there is always a path between the entity in the core knowledge and the entity in the hypothesis sentence. At the same time, the present invention regards the entities that appear in all hypotheses of the same problem as ordinary entities and gives them a lower weight; other entities that do not appear at the same time in the hypothesis are regarded as special entities and given a higher weight. The reachable score of the path between the entity from the core knowledge and the entity from the hypothesis sentence is calculated as follows, namely Among them, I(c i ,c j ) represents the symbolic function of whether entity i can reach entity j, and |path(i,j)| represents the length of the path passed by the two entities. h is a special entity set of hypothesis h, S(c j ,T h ) is a symbolic function of whether entity j belongs to a special entity set. To prevent division by zero, a small hyperparameter ε is added. is the i-th entity in the core knowledge sentence, is the jth entity in the hypothesis sentence.

[0041] (6) After obtaining the link relationship and corresponding scores between the entity set in the hypothesis sentence and the entity set in the core knowledge, the Kuhn-Munkres algorithm is used to find the optimal matching result of this weighted bipartite graph. The result is used as the hypothesis h i and core knowledge The matching score KM(h i ,e j ).

[0042] (7) The present invention performs a weighted summation of the context semantic retrieval score and the entity concept retrieval score, namely, st(h i ,e j )=γ·KM(h i ,e j )+(1-γ)·sim(h i ,e j), select the core knowledge with the highest score as the supporting evidence for the hypothesis, and use the score of this core knowledge as the confidence level of the hypothesis. Finally, select the hypothesis with the highest confidence level as the answer to the question. The present invention fine-tunes the model based on the cross-entropy loss function of the answer and the knowledge pool. That is, the loss function is:

[0043]

[0044] (8) Figure 4 As shown, in the multi-hop reasoning scenario, the present invention proposes a reasoning algorithm that combines contextual semantics and entity concept information. First, a dual-coding model combined with a masking mechanism is used to search for the core knowledge pool corresponding to the question. Then, the entity-aware knowledge verification module is used to select the most likely core knowledge to support the hypothesis. The fusion of the two-stage retrieval scores allows the model to self-select the most likely answer under the current parameters, and this is used as the basis for updating the model parameters.

[0045] (9) We conducted experiments on public datasets, and the experimental results demonstrated the effectiveness of our proposed two-stage knowledge retrieval model.

[0046] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A text multi-hop reasoning question answering method using two-stage retrieval, characterized by: The following steps are involved: S1, question q and each answer option a={a1,a2,...,a M } to obtain the hypothesis set h={h1,h2,...,h M }, where M is the number of options in the question. The original multiple-choice question, which simply concatenates question q and answer option a using a flag, is converted into a textual entailment question. S2. Divide the domain theoretical knowledge in the external text knowledge base into a core knowledge set E and a background knowledge set G, where the core knowledge e (e∈E) represents the core idea of the problem investigation, and the background knowledge g (g∈G) represents the auxiliary knowledge used to solve the problem; S3. In the first stage context-aware knowledge retriever module, the hypothesis h corresponding to the correct answer is i Correct Hypothesis and will support the correct assumption The established core knowledge e is called correct core knowledge will obscure any correct assumptions about the concept of partial entities. And the corresponding correct core knowledge Input into the dual encoder model in the context-aware knowledge retriever module; combined with the idea of contrastive learning, the correct hypothesis of other problems in the same batch training The corresponding correct core knowledge Negatively correlated knowledge After training, the correct hypothesis is obtained With correct core knowledge The mapping relationship between them; The step S3 comprises the following steps: S3.

1. Use the SpaCy text processing tool to extract all nouns and noun phrases from the background knowledge G as entities. For the noise extracted by the tool, semi-automatically reduce the noise by analyzing the entity occurrence frequency and entity meaning to form an entity vocabulary. S3.

2. Select the correct hypothesis based on the entity vocabulary Entities in the , and randomly mask a certain proportion of entities, will correctly assume Input the problem hypothesis encoder into the dual encoding model and simultaneously convert its corresponding correct core knowledge Input into the knowledge encoder in the dual encoder model, and calculate the distance between the two by cosine similarity; S3.

3. Correct Hypothesis for Other Problems in the Same Training Batch Corresponding core knowledge Negatively correlated knowledge Optimize the loss function to the correct core knowledge The negative log-likelihood value of , the loss function is as follows: S4, input all core knowledge e into the knowledge encoder in the dual encoding model, and output the semantic representation vector of the core knowledge e; then assume h i Input into the question hypothesis encoder in the dual encoder model to obtain the hypothesis h i The semantic representation vector of the core knowledge is retrieved by performing cosine similarity retrieval with the representation vectors of all core knowledge e to obtain the support hypothesis h i Candidate core knowledge pool The candidate core knowledge pool corresponding to question q At the same time, the core knowledge in the candidate core knowledge pool is obtained With the assumption h i The contextual semantic matching score between them; S5. In the second stage entity perception knowledge verification module, the reachability matrix between entities in the background knowledge set G is constructed, and the Kuhn-Munkres algorithm is used to verify the hypothesis h i Entities in the candidate knowledge pool and core knowledge in the candidate knowledge pool Calculate the reachability between entities and obtain the core knowledge e j and assuming h i Entity matching score between ; The step S5 includes: first extracting entities from the background knowledge G, and constructing a reachability matrix P between entities through the BFS algorithm, and then extracting core knowledge e from the candidate knowledge pool. j and assuming h i Entities in the calculation hypothesis h i Entities in the candidate knowledge pool and core knowledge e j The reachability score between entities is calculated as follows: Among them, I(c i ,c j ) represents the symbolic function of whether entity i can reach entity j, |path(i,j)| represents the length of the path passed by the two entities; T h is a set of special entities of hypothesis h, where special entities refer to entities that do not appear simultaneously in all hypotheses of the same problem, S(c j ,T h ) is a symbolic function of whether entity j belongs to a special entity set; to prevent division by 0, a hyperparameter ε is added; is the i-th entity in the core knowledge sentence, is the jth entity in the hypothesis sentence; after obtaining the reachability score between the entity set in the hypothesis sentence and the entity set in the core knowledge, the core knowledge e is obtained using the Kuhn-Munkres algorithm. j and assuming h i The score KM(h i ,e j ); S6. Perform a weighted sum of the scores obtained by the context-aware knowledge retriever module and the entity-aware knowledge verification module; for the i-th hypothesis h in the question q with M options i , the candidate core knowledge with the highest score Assuming h i The prediction core knowledge is used to predict the hypothesis with the highest score as the final answer to the question, and the dual encoder model is fine-tuned to minimize the loss value through the optimizer.

2. A method for multi-hop text reasoning question answering using two-stage retrieval according to claim 1, characterized in that: The step S1 of converting the multiple-choice question into a text-implied question includes: treating the core knowledge as a premise and the combination of the question and the answer options as a hypothesis; for interrogative sentence questions, using lexical, grammatical and dependency syntax analysis methods to form a semantically coherent and grammatically correct question hypothesis; for fill-in-the-blank questions, identifying the blanks and filling them with answer options.

3. The method for multi-hop text reasoning question answering using two-stage retrieval according to claim 1, characterized in that: In step S2, the core knowledge is a text expressing abstractions; the background knowledge includes a short description text including entity attributes, entity usage, and entity synonyms.

4. The method for multi-hop text reasoning question answering using two-stage retrieval according to claim 1, characterized in that: The process of acquiring the core knowledge pool in step S4 is as follows: all the hypotheses h for question q i All entities with the same position order are masked, and the cosine similarity is used to obtain the hypothesis h after each mask. i The top K core knowledge Right now Where L is the size of the knowledge base, sim(·) represents the function for calculating cosine similarity, top k {·} Take the K elements with the highest scores in the set; Then, for each hypothesis h i Candidate core knowledge Then sort and get the top K core knowledge corresponding to question q as the candidate knowledge pool for question q 5. The method for multi-hop text reasoning question answering using two-stage retrieval according to claim 1, characterized in that: The loss function of the dual encoder model in step S6 consists of two parts: the cross entropy loss based on the final answer and the cross entropy loss based on the knowledge pool, as follows: Loss=L local +L global Where N represents the sample size, M represents the number of options; y={y1,y2…,y M } is the one-hot representation of the answer label; p i represents the probability that the i-th choice is the correct answer, Represents the average score of all core knowledge corresponding to the i-th hypothesis in the core knowledge pool.

Images