A multiple-choice question-answering system based on first-order natural logic

Abstract

The present invention discloses a multiple-choice question-answering system based on first-order natural logic, including: a preprocessing module for converting questions and candidate answers into hypotheses; a first-order logic module for decomposing hypotheses into sub-hypotheses using first-order logic; a natural logic module for rewriting premises from an external corpus using natural logic to generate intermediate premises that convey the same meaning as the original premises, and iteratively generate reasoning paths from premises to hypotheses using a predefined maximum reasoning depth λ as a stopping criterion; a framework construction module for combining first-order logic with natural logic and integrating neural networks to construct a neural symbolic reasoning framework; a result output module for calculating the semantic similarity between intermediate premises and sub-hypotheses based on the neural symbolic reasoning framework, and outputting the final choice answer with the highest score according to the similarity score. The present invention not only shortens the reasoning path, but also enhances the alignment of verb phrases.

Description

Technical Field

[0001] The present invention belongs to the field of knowledge question answering, and in particular relates to a multiple-choice question answering system based on first-order natural logic. Background Art

[0002] Question answering is of great significance in the field of natural language processing. Many question answering tasks can be treated as textual entailment tasks, where the external corpus is treated as a collection of premises, while questions and candidate answers are combined as hypotheses to be evaluated. One of the challenges is that the external knowledge base does not explicitly answer all questions. Therefore, reasoning is needed to reveal many implicit answers. Moreover, in some cases, it is necessary to combine multiple pieces of knowledge to derive hypotheses.

[0003] Neural networks, especially pre-trained language models (PLMs) such as BERT, RoBERTa, and DeBERTa, have performed well in a variety of scenarios. However, a significant drawback of these models is the lack of the ability to trace back the decision process, that is, the lack of interpretability. As black boxes, they only provide decision results without explaining how these results were reached, making it difficult for people to understand and trust the model's decisions. To address this issue, a potential solution is to combine symbolic methods with neural networks. By integrating first-order logic into the reasoning process, the model can provide a step-by-step explanation for the decisions of the neural network. This approach aims to exploit the decomposition power of first-order logic while retaining the inherent robustness and high performance of neural networks.

[0004] Although first-order logic is very helpful for decomposing assumptions, there are some difficulties in the process of converting natural language text from external corpus to first-order logic. Take the sentence "loud noises can cause mammals to startle" as an example. When tuples are extracted for first-order logic reasoning, the final result is [loud noises, can cause, mammals], which does not capture important semantic information. On the other hand, natural logic uses deductive reasoning based on monotonicity in calculus, which enables it to reason directly on the surface of natural language text without converting to first-order logic. Therefore, it is of great significance to combine first-order logic and natural logic for problem solving to further enhance differentiable reasoning. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multiple-choice question-answering system based on first-order natural logic, comprising:

[0006] A preprocessing module to transform questions and candidate answers into hypotheses;

[0007] A first-order logic module, connected to the preprocessing module, for decomposing the hypothesis into sub-hypotheses using first-order logic;

[0008] A natural logic module, connected to the first-order logic module, for rewriting premises from an external corpus using natural logic to generate intermediate premises that convey the same meaning as the original premises, and iteratively generating a reasoning path from premises to hypotheses with a predefined maximum reasoning depth λ as a stopping criterion;

[0009] A framework construction module, connected to the natural logic module, for combining first-order logic with natural logic and integrating neural networks to construct a neural symbolic reasoning framework;

[0010] The result output module is connected to the framework construction module and is used to calculate the semantic similarity between the intermediate premise and the sub-hypothesis based on the neural symbolic reasoning framework, and output the final selected answer with the highest score according to the similarity score.

[0011] Preferably, the preprocessing module includes a task conversion unit and a fact pair retrieval unit;

[0012] The task conversion unit is used to convert questions and candidate answers into declarative sentences, treating the multiple-choice question-answering task as a textual entailment problem, and obtaining a target hypothesis Si;

[0013] The fact pair retrieval unit is used to retrieve relevant fact pairs based on the unstructured knowledge base K, and use the retrieved relevant fact pairs to perform correct option reasoning. The relevant fact pairs are represented as

[0014]

[0015] Preferably, the process of retrieving relevant fact pairs based on the unstructured knowledge base K includes:

[0016] Based on the target hypothesis Si as the starting point of the retrieval process, first retrieve fact f1 from the knowledge base, and then retrieve fact f2 based on the fact f1; wherein, the fact f2 contains at least one word from the set difference between the target hypothesis Si and the fact f1 and the set difference between the fact f1 and the target hypothesis Si; finally, select μ important fact pairs, and the value of μ is adjusted using the development set.

[0017] Preferably, the first-order logic module focuses on a subset of Horn clauses, in the following format:

[0018] p(X, Z)←p 1 (X,Y 1 )∧…∧p k (Y k , Z), (1)

[0019] Among them, P and P k is a predicate symbol, and X, Y kand Z is a variable symbol instantiated with a word or phrase;

[0020] The Horn clause includes a rule head p(X, Z) and a rule body p 1 (X, Y 1 )∧…∧p k (Y k , Z), the size of the rule body is represented by k.

[0021] Preferably, the first-order logic module includes a neural first-order reasoning unit and a logic reasoning unit;

[0022] The neural first-order reasoning unit is used to decompose the target hypothesis Si into triples by identifying the noun phrases present in the hypothesis based on the noun chunk extraction technology;

[0023] The logic reasoning unit is used to perform chain logic reasoning based on the neural network reasoning module, and uses the pre-trained language model DeBERTa as the model base to obtain the representation of each tag.

[0024] Preferably, the natural logic maps logical relationships according to the monotonicity determined by the context, and according to the monotonicity in calculus, monotonically does not change the logical relationships between words upward, and monotonically changes the logical relationships between words downward, to perform word-level insertion, deletion and replacement.

[0025] Preferably, the natural logic module comprises a neural natural logic reasoning unit;

[0026] The neural natural logic reasoning unit includes a candidate path generation subunit and a candidate path pruning subunit;

[0027] The candidate path generation subunit is used to generate a path that is closely consistent with the sub-hypothesis generated by first-order logic reasoning;

[0028] The pruning candidate path subunit is used to determine whether the replacement operation changes the semantics of the original premise through the monotonicity and logical relationship prediction module, and then filter out incorrect replacements.

[0029] Preferably, the formula expression of the semantic similarity is:

[0030]

[0031] where h lt and h rt Representation of subhypotheses using first-order logic, h v Representation of the intermediate premise using natural logic, V lt is the set of premises inferred from premise f1, V rtis the set of premises inferred from premise f2, the function sim(·) is used to calculate the semantic similarity, and the sigmoid function σ is used to convert the score into probability. represents the probability of the positive class, represents the probability of the negative class.

[0032] Compared with the prior art, the present invention has the following advantages and technical effects:

[0033] The framework proposed in the present invention adopts a two-way reasoning process. First, the question and candidate answers are converted into hypotheses, and the hypotheses are decomposed into sub-hypotheses using first-order logic. At the same time, natural logic is used to rewrite the premise from an external corpus to generate an intermediate premise that conveys the same meaning as the original premise, and multiple reasoning paths are iteratively generated. When the answer is finally selected, the semantic similarity between the intermediate premise and the sub-hypothesis is calculated using the similarity score, and the final selected answer with the highest score is output based on the similarity score. The framework seamlessly integrates the advantages of neural networks and systems based on first-order logic and natural logic, and can enhance reasoning in a more efficient way.

[0034] The present invention integrates first-order logic and natural logic to promote explainable reasoning. This integration realizes an orderly reasoning process and decomposes complex hypotheses into tractable sub-hypotheses. In addition, this integration ensures that the semantic information of external knowledge is retained throughout the reasoning process. The phrase-level replacement technology is introduced in the natural logic reasoning process, which not only shortens the reasoning path but also enhances the alignment of verb phrases.

[0035] The proposed framework is comprehensively evaluated on three benchmark datasets, and the evaluation results show that the proposed framework outperforms existing methods and achieves state-of-the-art performance. In addition, the framework also provides a reasonable reasoning path. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 This is an architectural diagram of the FINAL framework of an embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] like Figure 1 As shown, this embodiment provides a multiple-choice question-answering system based on first-order natural logic, including:

[0041] A preprocessing module to transform questions and candidate answers into hypotheses;

[0042] A first-order logic module, connected to the preprocessing module, is used to decompose the hypothesis into sub-hypotheses using first-order logic;

[0043] The natural logic module is connected to the first-order logic module and is used to rewrite the premise from the external corpus using natural logic to generate an intermediate premise that conveys the same meaning as the original premise, and iteratively generates the reasoning path from premise to hypothesis with a predefined maximum reasoning depth λ as the stopping criterion;

[0044] The framework building module is connected with the natural logic module to combine the first-order logic with the natural logic and integrate the neural network to build a neural symbolic reasoning framework;

[0045] The result output module is connected to the framework building module and is used to calculate the semantic similarity between the intermediate premise and the sub-hypothesis based on the neural symbolic reasoning framework, and output the final selected answer with the highest score according to the similarity score.

[0046] Further, the preprocessing module includes a task conversion unit and a fact pair retrieval unit;

[0047] The task conversion unit is used to convert questions and candidate answers into declarative sentences, treating the multiple-choice question-answering task as a textual entailment problem and obtaining the target hypothesis Si;

[0048] The fact pair retrieval unit is used to retrieve relevant fact pairs based on the unstructured knowledge base K, and use the retrieved relevant fact pairs to perform correct option reasoning. The relevant fact pairs are represented as

[0049]

[0050] Furthermore, the process of retrieving relevant fact pairs based on the unstructured knowledge base K includes:

[0051] Based on the target hypothesis Si as the starting point of the retrieval process, first retrieve fact f1 from the knowledge base, and then retrieve fact f2 based on fact f1; wherein fact f2 contains at least one word from the set difference between the target hypothesis Si and fact f1 and the set difference between fact f1 and the target hypothesis Si; finally, select μ important fact pairs, and the value of μ is adjusted using the development set.

[0052] Furthermore, the first-order logic module focuses on a subset of Horn clauses, which have the following format:

[0053] p(X, Z)←p 1 (X, Y 1 )∧…∧p k (Y k , Z), (1)

[0054] Among them, P and P k is a predicate symbol, and X, Y k and Z is a variable symbol instantiated with a word or phrase;

[0055] The Horn clause consists of a rule head p(X, Z) and a rule body p 1 (X, Y 1 )∧…∧p k (Y k , Z), the size of the rule body is represented by k.

[0056] Furthermore, the first-order logic module includes a neural first-order reasoning unit and a logic reasoning unit;

[0057] The neural first-order reasoning unit is used to decompose the target hypothesis Si into triples based on the noun chunk extraction technique by identifying the noun phrases present in the hypothesis;

[0058] The logical reasoning unit is used to perform chain logical reasoning based on the neural network-based reasoning module and uses the pre-trained language model DeBERTa as the model base to obtain the representation of each token.

[0059] Furthermore, natural logic maps logical relationships according to the monotonicity determined by the context, and according to the monotonicity in calculus, performs word-level insertion, deletion, and replacement by monotonically not changing the logical relationships between words upward and monotonically changing the logical relationships between words downward.

[0060] Further, the natural logic module includes a neural natural logic reasoning unit;

[0061] The neural natural logic reasoning unit includes a candidate path generation subunit and a candidate path pruning subunit;

[0062] The candidate path generation subunit is used to generate paths that are closely consistent with the sub-hypotheses generated by first-order logic reasoning;

[0063] The pruning candidate path subunit is used to determine whether the replacement operation changes the semantics of the original premise through the monotonicity and logical relationship prediction modules, thereby filtering out incorrect replacements.

[0064] Furthermore, the formula for semantic similarity is:

[0065]

[0066] where h lt and h rt Representation of subhypotheses using first-order logic, h v Representation of the intermediate premise using natural logic, V lt is the set of premises inferred from premise f1, V rt is the set of premises inferred from premise f2, the function sim(·) is used to calculate the semantic similarity, and the sigmoid function σ is used to convert the score into probability. represents the probability of the positive class, represents the probability of the negative class.

[0067] Furthermore, this embodiment aims to combine first-order logic with natural logic to enhance the multiple-choice question-solving process. Specifically, a model is developed to treat the multiple-choice question-answering task as a textual entailment problem, and accurately identify the correct answer to the multiple-choice question based on an unstructured knowledge base. First, the question and the eight candidate answers are converted into declarative sentences, called target hypotheses Si, where i ranges from 1 to 8. The next step involves retrieving a set of relevant fact pairs from the unstructured knowledge base K, represented as The purpose is to use the retrieved facts to infer the correct option. To achieve this goal, the method of this embodiment focuses on adopting a neural symbolic approach. The model uses first-order logic to decompose assumptions and adopts natural logic as the underlying proof to facilitate the construction of the proof process.

[0068] This example uses first-order logic as a symbolic method in a neural symbolic framework. Within this framework, this example focuses on a subset of Horn clauses, which have the following format:

[0069] p(X, Z)←p 1 (X, Y1)∧…∧p k (Y k , Z), (1)

[0070] Among them, P and P k is a predicate symbol, and X, Y k and Z are variable symbols instantiated with words or phrases. A Horn clause consists of a rule head (denoted as p(X, Z)) and a rule body (denoted as p 1 (X, Y 1)∧…∧p k (Y k , Z)), and the size of the rule body is represented by k.

[0071] In this embodiment, an atom refers to a predicate and its parameters. For example, p(X, Z) is considered an atom, which can be instantiated into the atom can_cause(static electricity, a forest fire). Here, variables X and Z are assigned instance values ​​"staticelectricity" and "aforest fire" respectively. The predicate p is assigned the instance value "can_cause".

[0072] Natural logic is a formal proof theory that allows reasoning directly on the surface of natural language. It is based on the monotonicity principle. In natural logic proofs, the main operations involve word-level insertion, deletion, and substitution, which are based on contextual monotonicity. The core reasoning rules of natural logic rely on word-level substitution and monotonicity between contexts. As shown in Table 1, seven semantic relations are proposed, namely For example, replacing the word "plants" with "shrubs" corresponds to the reverse implication relation, that is, Natural logic then maps logical relations according to the monotonicity determined by the context. According to the monotonicity in calculus, upward monotonicity does not change the logical relations between words, while downward monotonicity does change the logical relations between words. For example, the quantifier "all" is downward monotonic in its first argument. Therefore, given It can be inferred (such as the sentence ).

[0073] Table 1

[0074]

[0075] like Figure 1 As shown, this embodiment proposes a neural symbolic framework called FINAL, which combines first-order logic with natural logic to achieve differentiable reasoning. The framework rewrites external knowledge by applying first-order logic decomposition assumptions and natural logic to perform pre-defined bidirectional reasoning. The iterative process is performed in the natural logic module with a predefined maximum reasoning depth λ as the stopping criterion. After meeting this criterion, this embodiment evaluates the semantic similarity between the sub-hypothesis of the first-order logic decomposition and the intermediate premise inferred by natural logic.

[0076] Knowledge selection: Starting from a question sentence (e.g., "What can cause a forest fire") and a candidate answer (e.g., "static electricity"), FINAL transforms them into a declarative hypothesis sentence Si, in this case, "static electricity can cause a forest fire".

[0077] Answering combinatorial questions requires collecting information from multiple external knowledge. This requires retrieving relevant fact pairs from the external knowledge base K. Assume Si as the starting point for FINAL to initiate the retrieval process. In order to retrieve relevant fact pairs, a two-step information retrieval (IR) process is required. The first step involves retrieving fact f1 from the knowledge base. Subsequently, fact f2 is retrieved based on fact f1. It is worth noting that the retrieved fact f2 must contain at least one word from the set difference of Si and f1 and the set difference of f1 and Si. Finally, μ important fact pairs are selected based on the sum of the individual IRscores. The value of μ is adjusted using the development set.

[0078] Neural first-order reasoning;

[0079] Triple extraction: First, the model converts the hypothesis Si into a triple format, which is based on the noun chunk extraction technique by identifying the noun phrases present in the hypothesis. For example, for the hypothesis "static electricity may cause forest fires", this sentence contains two noun phrases, namely "static electricity" and "forest fires". The rest of the sentence can be regarded as the predicate. Therefore, the hypothesis is represented in triple format as [static electricity may cause, forest fires].

[0080] First-order logic module: This embodiment uses the following rules to decompose the hypothesis into triples:

[0081] p 1 (X, Z)←p 2 (X, Y)∧p 3 (Y, Z), (2)

[0082] where X, Y, and Z are variables that can be instantiated with words or phrases. nRepresents the predicate. In this embodiment, the rule body consists of two atoms. The method of this embodiment can be applied to rules containing any number of atoms in the rule body. When the model is decomposed, two new tuples p2(X, Y) and p3(Y, Z) are generated. Intuitively, these new tuples inherit parameters X and Z from the rule head, and they are connected by a shared parameter Y. Therefore, the model only needs to predict three tags-two predicates and one shared parameter. For example, this example [static electricity, may lead to, forest fire]. This can be decomposed into two new sub-hypotheses: [static electricity, predicate 2, entity ★] and [entity ★, predicate 3, forest fire]. Inspired by neural theorem proof, this embodiment introduces a neural network-based reasoning module for chain logic reasoning. In order to obtain the representation of each tag, the pre-trained language model DeBERTa is used as the model base. For all input tags Si=([CLS], w1, ..., wL), the hidden state Henc=(h[CLS], h1, ..., hL) is calculated using the following equation:

[0083] H enc =DeBERTa(φ emb (S i )), (3)

[0084] Where L represents the number of tokens in the hypothesis, Refers to the embedding layer of DeBERTa.

[0085] The vector representation of the two predicates is calculated as follows:

[0086]

[0087] where h p1 、h p2 and h p3 are the vector representations of predicate 1, predicate 2, and predicate 3, respectively. h[CLS] refers to the vector representation of the entire sentence. avg(·,·) represents the mean function. Parameter W p2 , W p3 、b p2 and b p3 is learnable, SiLU is the activation function. Shared parameter h entity The vector representation of is calculated as follows:

[0088]

[0089] where h X 、h p1 and h Z are vector representations of parameter X, predicate 1, and parameter Z, respectively. The function avg(·,·,·,·) calculates the average value of the input vector. Parameter W e and b eare learnable parameters. Note that the vector h entity Represents an entity in the embedding space. By calculating the Euclidean distance decoding entity with the predefined entity set, after obtaining the vector representation of the generated predicate and shared parameters, the model can calculate the vector representation of two new tuples. For convenience, this embodiment refers to these two tuples as left subhypothesis hyp l and the right hypotheses hyp r The representations of these two sub-hypotheses are calculated as follows:

[0090]

[0091] where h lt and h rt are two generated tuples p 2 (X, Y) and p 3 Vector representation of (Y, Z).

[0092] Neural natural logic reasoning;

[0093] Candidate path generation: This embodiment uses a forward proof process to generate paths that are closely consistent with the sub-hypotheses generated by first-order logic reasoning. The method of this embodiment follows natural logic reasoning steps to ensure that the original premise implies the intermediate premise. Once the semantic proximity of the sub-hypothesis to the intermediate premise is established, the model can prove that the hypothesis can be implied by the fact pair through reasoning steps.

[0094] For convenience, the following describes how to get from fact 1 The same reasoning method can be applied to the fact 2 to the right sub-hypothesis. The model uses spaCy to obtain part-of-speech (POS) tags. When making replacements, the framework FINAL ignores tokens with the following part-of-speech tags: prepositions, determiners, coordinating conjunctions, cardinal numbers, personal pronouns, and modal verbs. The phrase-level reasoning process is beneficial for shortening the reasoning path. In addition, some predicates take the form of phrases. Therefore, the model utilizes BART for decoding, which uses seq2seq to decode one or more tokens for each mask.

[0095] The model starts reasoning from fact f1, which can be regarded as a premise. This fact consists of L words, represented as To perform mask filling, the words in the premise are masked and then input into BART. In order to narrow the semantic distance to the left sub-hypothesis, reasoning is also performed through insertion and deletion operations, and the model only inserts or deletes adjectives or adjective phrases. The insertion operation uses a masking mechanism by inserting a [MASK] tag in front of the noun. The model tracks the positions of insertion / replacement / deletion to avoid repeated operations at the same position. The list of candidate words or phrases is obtained through the output of BART. The method of this embodiment takes into account the context instead of searching directly from WordNet.

[0096] Pruning candidate paths: To ensure that the intermediate premise can be deduced from the original premise, it is important to follow natural logical reasoning rules. This involves using monotonicity and logical relationship prediction modules to determine whether the replacement operation changes the semantics of the original premise. By doing so, any incorrect replacements can be filtered out.

[0097] To determine the monotonicity of each word, this embodiment uses the natlog parser. This parser can obtain the monotonicity of words. To predict logical relationships, this embodiment uses the fine-tuned RoBERTa model. By replacing the original tag w l and the resulting phrase or token c l Input into the RoBERTa model, the probability distribution P of the seven natural logical relations given in Table 1 can be calculated. r The calculation formula is as follows:

[0098]

[0099] Where f(·) represents the RoBERTa model. The input format of this model is [CLS]w l [SEP]c l [SEP]. Here, w l refers to the original markup, while c l Represents the generated phrase or token. The logical relation with the highest score max(P r ) is considered as a predictive logical relationship.

[0100] According to the word w l The monotonicity of the premise and the logical relationship between the replacement words are used to derive the logical relationship between the sentences before and after the replacement using the projection function. When the monotonicity is upward monotonic, the sentence-level logical relationship is consistent with the logical relationship between the replacement words. However, if the monotonicity is downward monotonic, the projection operator ξ can be used to predict the sentence-level logical relationship, as shown in Table 2. In order to ensure that the semantics of the original premise remains unchanged, the model only performs equivalence ≡ and forward implication. We can reason about sentence-level relations of

[0101] Table 2

[0102]

[0103] One way to train the natural logic module is through end-to-end training. However, this approach may result in an overly large reasoning search space and low efficiency. As a solution, this embodiment preprocesses the natural logic module, which helps reduce the complexity of the reasoning process in the final framework. The model performs phrase-level insertion, substitution, and deletion in a depth-first manner to expand the proof process until the maximum depth λ is reached.

[0104] In order to evaluate the semantic similarity between the sub-hypotheses decomposed by first-order logic and the intermediate premises obtained by natural logic reasoning, this embodiment adopts the DeBERTa model to learn this semantic proximity. The validity of the proof depends on the semantic similarity calculated on any branch of the sentences derived from first-order logic and natural logic. The semantic similarity is calculated as follows:

[0105]

[0106] where h lt and h rt Represents the representation of subhypotheses using first-order logic. h v Representation of intermediate premises using natural logic. V lt is the set of premises inferred from premise f1, V rt is the set of premises inferred by premise f2. The function sim(·) is used to calculate the semantic similarity. The sigmoid function σ is used to convert the score into probability. represents the probability of the positive class, represents the probability of the negative class. In this case, the semantic similarity can be interpreted as the support probability of the fact pair (f1, f2), which corresponds to the probability of the corresponding answer. To determine the correct answer among all candidate options, apply the same steps and select the one with the highest score Finally, the answer corresponding to the hypothesis with the highest score is selected.

[0107] This example uses the question and each option to construct N different input sequences. The model considers the input sequence corresponding to the standard answer as the positive class, and all other sequences as the negative class. For each input sequence, the model obtains the probabilities of the negative class and the positive class, which are expressed as and Therefore, the loss function takes the following form:

[0108]

[0109] in and Represents the category label. When the i-th candidate answer is correct, otherwise,

[0110] Below is the experimental setup, including benchmark datasets, evaluation metrics, baseline models, and implementation details.

[0111] Benchmark datasets: This example conducts experiments on different datasets that require retrieving facts from external corpora to answer multiple-choice questions. Table 3 shows the statistics of the three datasets used in the experiment: QASC, WorldTree, and Wikihop.

[0112] Table 3

[0113]

[0114] QASC is a multiple-choice science question answering dataset that tests common sense and scientific reasoning. It consists of 9,980 questions with 1 correct answer and 7 distractor answers, of which 4 candidate answers are strong adversarial distractor answers. To answer these questions, the model needs to retrieve knowledge from a knowledge base containing 17.2 million facts. WorldTree focuses on multiple-choice science question answering. It consists of 2,459 questions and four options, of which only one option is correct. The dataset is divided into Challengeset and Easyset. To answer these questions, the model needs to use a knowledge base containing a large amount of common sense and scientific facts to build explanations. WikiHop is a multi-hop reading comprehension dataset. In this dataset, it is necessary to combine information from multiple documents to derive the correct answer. Each question contains a query of the form p(e, X), where e is an entity, X is the entity that needs to be predicted from a list of candidate entities C, and p is the query predicate. The 4 domain-specific datasets are publishers, developers, cities, and record labels. For each domain, there is a training set and a development set with different numbers of questions. Since the test set of WikiHop is not public, this embodiment uses the development set for evaluation.

[0115] Evaluation Metrics: Accuracy was used as the evaluation metric for all datasets to evaluate the performance of the model. To evaluate the quality of the explanations, a manual evaluation was performed. Three graduate students proficient in English annotations were responsible for scoring 0 or 1 to evaluate the credibility of the provided reasoning path. Each reasoning path was scored independently by all three raters. The average interpretability score is expressed as Determined by the following formula:

[0116]

[0117] Where M represents the total number of random samples. represents the majority annotation of the qth sample. is the indicator function, when 1 if the value is 0, otherwise it is 0. It refers to the annotation provided by the u-th rater for the q-th sample, and its value is 0 or 1.

[0118] To measure the quality of the interpretability scores annotated by human annotators, the inter-rater agreement was calculated using Fleiss’ Kappa statistic. The calculation of the kappa value κ is based on the following formula:

[0119]

[0120] Where P represents the proportion of agreement between evaluators. e represents the expected agreement ratio under random annotation conditions. M is the total number of random samples, U represents the total number of raters, J represents the total number of manually annotated categories, which is binary in this case, represented by values ​​0 and 1, and n qj represents the number of raters who assigned the qth sample to the manually labeled category j.

[0121] To conduct baseline experiments, this embodiment uses two types of large language models (LLMs): LLMs accessed through the API and open source LLMs. In the first category, GPT-3.5 and GPT-4 are called through the OpenAI API. Specifically, versions gpt-3.5-turbo-1106 and gpt-4-1106-preview are used. For the second category, this embodiment utilizes open source models, including LLAMA2, Vicuna, ChatGLM, and Baichuan. To ensure fair evaluation, this embodiment uses 7B and 13B versions of LLM, and the model size is comparable to the number of parameters of the model proposed in this embodiment. In addition, this embodiment focuses on the chat versions of these models because they have been fine-tuned for chat-style use cases. Throughout the experiment, the hyperparameter settings recommended for all LLMs were followed, and the temperature was 0 to ensure reproducibility.

[0122] Experiments were conducted on the Wikihop dataset using the same prompts as the QASC and WorldTree datasets, however, the experimental results were not satisfactory. We conjecture that the Wikihop dataset requires specific knowledge, so we designed prompts by adding knowledge as prompts. In addition, in order to better understand the triple format of the query, we separated the query and the candidate.

[0123] Fine-tuned models: In addition to the large language model, several fine-tuned models trained on the target dataset are compared. Specifically, the BERT Large, RoBERTa Large, and DeBERTa Large models are utilized. For these models, the input format consists of context. i ) and the question in series, followed by the answer (answeri ), all content is enclosed in special tags.

[0124] During the inference process, the model performs i ) normalizes the output scores of and selects the answer with the highest score. In addition, a strong baseline model called TEAM is included, which treats the task of multiple-choice answering as a binary classification problem. The TEAM model classifies (question, correct answer) as a positive instance and (question, incorrect answer) as a negative instance. TEAM achieves significant improvements on multiple datasets and is based on the "DeBERTa-large" model.

[0125] In addition, this example compares FINAL with two neural symbolic baselines: NeuNLI and DILR. The NeuNLI model incorporates natural logic reasoning into the deep learning architecture to enhance question answering performance, while the DILR model first extracts query-related information and then uses first-order logic for logical reasoning. It is worth mentioning that the DILR model is the best performing model on the WikiHop dataset.

[0126] The model uses the "DeBERTa-large" pre-trained language model with a vector dimension of 1024. In the reasoning process based on natural logic, the maximum search depth is set to 7 and the limit is 4 related fact pairs. The implementation of the model and training algorithm is based on the Transformers library. The model is fine-tuned using the AdamW optimizer with an initial learning rate of 1e-6. The training uses 50 epochs using an A100 GPU with a total of 80G memory. For the QASC and WorldTree datasets, the model with the best performance on the development set is selected for testing on the test set.

[0127] This example evaluates the performance of FINAL through the above experimental settings and comprehensively analyzes the results obtained. In addition, the impact of hyperparameters μ and λ is explored, and a case analysis with an interpretable path is given. The experimental analysis results are as follows:

[0128] When comparing NeuNLI and FINAL, it is clear that our approach performs better, improving by 2.18% on QASC, 2.24% on WorldTree, and 2.66% on WikiHop. This improvement is attributed to the model’s use of first-order logic to decompose assumptions, which is beneficial for questions that require the composition of multiple sentences.

[0129] The method in this example is 2.81% better than the DILR-DeBERTa method on the WikiHop dataset. This demonstrates the superior performance of the model, which incorporates a natural logic module. This module helps to simplify unstructured natural language, thereby improving consistency with first-order logic reasoning results.

[0130] The method of this embodiment also surpasses the performance of TEAM, improving by 0.98% on QASC, 3.04% on WorldTree, and 4.24% on WikiHop. Both methods combine contextual semantic information with DeBERTa in question answering and treat the task as a binary classification task. However, the method of this embodiment introduces first-order logic and natural logic, which not only improves the performance, but also provides a reasonable explanation.

[0131] On the WorldTree dataset, the model achieved 91.97% on simple datasets and 81.70% on challenging datasets compared to other open source models. This demonstrates extraordinary capabilities in understanding and reasoning. It is worth noting that both GPT-3.5 and GPT-4 showed higher accuracy than other models, highlighting the excellent performance of large language models in common sense reasoning tasks.

[0132] It is worth noting that GPT-3.5 and GPT-4 achieve higher accuracy on the WorldTree dataset, 93.10% and 94.07%, respectively. In contrast, their performance on the QASC dataset is relatively weak. It is speculated that this difference may be the uniqueness of the QASC dataset itself, which has 4 adversarial choices. These adversarial choices introduce greater uncertainty in the prediction, which poses a challenge to large language models because large language models often swing between the top 2 or top 3 maximum prediction probabilities. However, the method of this embodiment achieves state-of-the-art performance on the QASC dataset, overcoming the challenges brought about by the presence of adversarial choices.

[0133] In addition, the DILR-DeBERTa method performs poorly on the QASC and WorldTree datasets. This suggests that the DILR-DeBERTa method is more suitable for well-structured queries, but may not be suitable for natural language questions. In contrast, the method of this embodiment performs well on all three datasets. The experimental results on the WikiHop dataset are consistent with those on QASC and WorldTree, highlighting the versatility and robustness of the model, and the framework may be better adapted to various data patterns.

[0134] To evaluate the impact of different modules, this example performs an ablation study on the test sets of QASC and WorldTree and the development set of WikiHop. The study consists of three variants of the model: 1) FINAL (without first-order logic): This variant uses only the natural logic module for reasoning and does not include the first-order logic module. 2) FINAL (without natural logic): This variant uses only the first-order logic module for reasoning and does not include the natural logic module. 3) FINAL (word level): This variant retains the first-order logic module and the natural logic module for reasoning. However, in the natural logic module, word-level replacement is used instead of phrase-level replacement. The experimental conclusions are as follows:

[0135] Effectiveness of the first-order logic based module. Comparing FINAL with FINAL(without first-order logic), the performance of FINAL(without first-order logic) drops significantly. The average accuracy score on the WikiHop development set drops from 87.80% to 84.24%. The same trend can be observed in the other two datasets. This suggests that some questions need to consist of multiple sentences. The alignment of triples obtained with the first-order logic module serves as a guiding force to ensure that the inferred information is not only semantically meaningful but also logically coherent.

[0136] Based on the effectiveness of the natural logic module. Comparing the performance of FINAL with and without the natural logic module, including the module significantly improves the performance (3.26% on QASC, 2.80% on WorldTree, and 3.21% on WikiHop). This finding highlights the valuable role of natural logic in facilitating the reasoning process, especially in simplifying complex and unstructured natural language into more concise and easy-to-process natural language. Experimental results show that the first-order logic module and the natural logic module play an equally important role in the model.

[0137] Effectiveness of phrase-level natural logic reasoning. There is a significant drop in performance when using word-level natural logic comparison (2.07% on QASC, 1.76% on WorldTree, and 1.98% on WikiHop). This shows that using phrase-level natural logic is beneficial for reasoning, indicating that phrase-level substitution is beneficial for verb phrases and reduces redundant expansion.

[0138] For comparison, this example compares the results of the TEAM method, which achieves the second best performance on this dataset. The results show that even on the basic model RoBERTa-large, the method of this example outperforms the TEAM method. The experimental results consistently show that the method of this example based on the DeBERTa or RoBERTa model always outperforms the respective TEAM method. This shows that the integration of symbolic methods helps improve performance.

[0139] In order to analyze the impact of different hyperparameters on model performance, a series of experiments were conducted on the development set of the WorldTree dataset.

[0140] Effect of the number of fact pairs (μ). As the number of related fact pairs increases from 1 to 4, the model’s accuracy remains at about 93%, indicating that even a pair of related facts obtained from the retrieval module contains sufficient knowledge for reasoning. In addition, the model demonstrates effective utilization of a small amount of external knowledge. It is worth noting that adding more external knowledge pairs does not significantly improve the accuracy. In fact, increasing the number of knowledge pairs may introduce redundant information rather than providing valuable insights.

[0141] The impact of the number of reasoning steps (λ) in the natural logic module. As the number of reasoning steps increases from 1 to 4, the accuracy of the model improves significantly. This can be attributed to the fact that the inferred sentences are more consistent with the triples obtained from the first-order logic module. However, when more than 4 reasoning steps are added, the accuracy starts to decrease. This decrease is due to the potential divergence from the original semantics caused by the increase in the number of reasoning steps, resulting in semantic drift.

[0142] Below we analyze the distribution of FINAL's reasoning steps on the WorldTree dataset. More than three-quarters of the questions are predicted with at least one step, and more than a quarter of the questions are predicted with multiple steps. The threshold of four reasoning steps appears to be effective, providing sufficient reasoning for the task while avoiding unnecessary excessive reasoning steps. Most questions are affected by one step of reasoning, which shows that for a large part of the test set, a single reasoning step is sufficient for the model to make accurate predictions. In addition, the percentage of correctly predicted questions at different reasoning steps is evaluated. Regardless of the number of reasoning steps, about 90% of the questions can be correctly predicted, which confirms the effectiveness of the method.

[0143] In the analysis of the natural logic module, the distribution of word-level and phrase-level substitutions was examined. The results show that natural logic reasoning mainly occurs at the phrase level. Specifically, it accounts for 61.60% on the QASC dataset, 75.81% on the WorldTree dataset, and almost exceeds 85% on the Wikihop dataset. These results show that the key factor in understanding and reasoning about external knowledge using natural logic is to focus on phrase-level information.

[0144] To evaluate the interpretability of the models, quantitative evaluation was performed through manual evaluation. NeuNLI was set as the baseline and significance was tested using a paired t-test at a significance level of 0.05. To ensure a fair comparison, 100 questions correctly classified by NeuNLI and FINAL were randomly selected from the WorldTree dataset.

[0145] The results of Fleiss' Kappa values ​​and average interpretability scores show significant differences less than 0.05. According to commonly used interpretation criteria, Fleiss's NeuNLI Kappa value (0.59) reaches moderate consistency (≥0.41), and FINAL value (0.67) reaches substantial consistency (≥0.61). These results emphasize the quality of manual annotations. The average interpretability score of FINAL is significantly higher than that of NeuNLI, and all annotators agree on the improvement of FINAL. This improvement can be attributed to the ability of the method of this embodiment to decompose complex hypotheses through first-order modules, and the use of phrase-level substitutions during reasoning.

[0146] The following provides an illustrative example to demonstrate the effectiveness of the method. The proposed hypothesis states that "seeing whether bone has been damaged is used in medical technology". After being processed by the first-order logic module, two tuples are derived: [bone has been damaged, need, rays] and [rays, used in, technology]. The shared parameter between the two tuples is "rays". Using external knowledge, the fact pairs obtained by the retrieval module are (("x-rays are used to see whether bone has been damaged or destroyed", "x-rays are used in medical technology”). Analyzing the reasoning path, the left path shows the natural logic reasoning steps, where FINAL converts "see" to "determine" and adds "bone" before "x-rays" based on the context. On the other hand, the reasoning path on the right shows the step of removing the adjective "medical". Ultimately, the outputs of the two logic modules convey similar semantics, and FINAL successfully identifies the key shared parameter "rays". In contrast, the result of GPT-3.5 is inaccurate. It predicts the answer "(A)crucial" on the grounds that "the word 'crucial' fits the context of determining whether bones are suitable for medical technology because it implies the importance and necessity of the evaluation." This indicates a misunderstanding of the context. This example gives examples of incorrect predictions by GPT-3.5 and GPT4 on the QASC dataset. GPT4 predicts "alligators", which is an illusion. Although alligators have jaws, they do not have multiple rows of teeth. The fact is that alligators grow new teeth, but they do not have multiple rows of teeth.

[0147] In the field of natural language reasoning, researchers have explored the integration of natural logic and neural networks to enhance reasoning capabilities. Although these methods aim to directly reason about natural language, they encounter challenges in processing knowledge combinations under the constraints of natural logic rules. To address these challenges, this embodiment regards the question-answering task as a textual entailment task and introduces a neural symbolic framework. Differentiable reasoning is promoted by combining first-order logic and natural logic. By leveraging the decomposition properties of first-order logic and the reasoning power of natural logic based on monotonicity, the model attempts to overcome the limitations observed in existing methods. In addition, there are studies that delve into unsupervised iterative techniques for question answering, mainly emphasizing retrieval. In contrast, the research in this embodiment focuses on the reasoning part, especially on clarifying the explainable reasoning process. Through experiments on three different datasets, the experimental results demonstrate that the framework proposed in this embodiment has superior performance compared with existing methods.

[0148] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multiple-choice question-answering system based on first-order natural logic, characterized in that: include: A preprocessing module to transform questions and candidate answers into hypotheses; A first-order logic module, connected to the preprocessing module, for decomposing the hypothesis into sub-hypotheses using first-order logic; A natural logic module, connected to the first-order logic module, for rewriting premises from an external corpus using natural logic to generate intermediate premises that convey the same meaning as the original premises, and iteratively generating a reasoning path from premises to hypotheses with a predefined maximum reasoning depth λ as a stopping criterion; A framework construction module, connected to the natural logic module, for combining first-order logic with natural logic and integrating neural networks to construct a neural symbolic reasoning framework; A result output module, connected to the framework building module, is used to calculate the semantic similarity between the intermediate premise and the sub-hypothesis based on the neural symbolic reasoning framework, and output the final selected answer with the highest score according to the similarity score; The preprocessing module includes a task conversion unit and a fact pair retrieval unit; The task conversion unit is used to convert questions and candidate answers into declarative sentences, treating the multiple-choice question-answering task as a textual entailment problem, and obtaining a target hypothesis Si; The fact pair retrieval unit is used to retrieve relevant fact pairs based on the unstructured knowledge base K, and use the retrieved relevant fact pairs to perform correct option reasoning. The relevant fact pairs are represented as The process of retrieving relevant fact pairs based on the unstructured knowledge base K includes: Based on the target hypothesis Si as the starting point of the retrieval process, first retrieve fact f1 from the knowledge base, and then retrieve fact f2 based on the fact f1; wherein, the fact f2 contains at least one word from the set difference between the target hypothesis Si and the fact f1 and the set difference between the fact f1 and the target hypothesis Si; finally, select μ important fact pairs, and the value of μ is adjusted using the development set.

2. The multiple-choice question-answering system based on first-order natural logic according to claim 1, characterized in that: The first-order logic module focuses on a subset of Horn clauses, which have the following format: p(X,Z)←p1(X,Y1)∧…∧p k (Y k ,Z), (1) Among them, P and P k is a predicate symbol, and X, Y k and Z is a variable symbol instantiated with a word or phrase; The Horn clause includes a rule head p(X, Z) and a rule body p1(X, Y1)∧…∧p k (Y k , Z), the size of the rule body is represented by k.

3. The multiple-choice question-answering system based on first-order natural logic according to claim 1, characterized in that: The first-order logic module includes a neural first-order reasoning unit and a logic reasoning unit; The neural first-order reasoning unit is used to decompose the target hypothesis Si into triples by identifying the noun phrases present in the hypothesis based on the noun chunk extraction technology; The logic reasoning unit is used to perform chain logic reasoning based on the neural network reasoning module, and uses the pre-trained language model DeBERTa as the model base to obtain the representation of each tag.

4. The multiple-choice question-answering system based on first-order natural logic according to claim 1, characterized in that: The natural logic maps logical relationships according to the monotonicity determined by the context, and according to the monotonicity in calculus, performs word-level insertion, deletion and replacement by monotonically not changing the logical relationships between words upward and monotonically changing the logical relationships between words downward.

5. The multiple-choice question-answering system based on first-order natural logic according to claim 1, characterized in that: The natural logic module includes a neural natural logic reasoning unit; The neural natural logic reasoning unit includes a candidate path generation subunit and a candidate path pruning subunit; The candidate path generation subunit is used to generate a path that is closely consistent with the sub-hypothesis generated by first-order logic reasoning; The pruning candidate path subunit is used to determine whether the replacement operation changes the semantics of the original premise through the monotonicity and logical relationship prediction module, and then filter out incorrect replacements.

6. The multiple-choice question-answering system based on first-order natural logic according to claim 1, characterized in that: The formula expression of the semantic similarity is: where h lt and h rt Representation of subhypotheses using first-order logic, h v Representation of the intermediate premise using natural logic, V lt is the set of premises inferred from premise f1, V rt is the set of premises inferred from premise f2, the function sim(·) is used to calculate the semantic similarity, and the sigmoid function σ is used to convert the score into probability. represents the probability of the positive class, represents the probability of the negative class.

Images