A multi-hop question answering framework based on label smoothing
By employing a tag-smoothing multi-hop question answering framework, combining coarse and fine screening to select the optimal article pairing, and using the F1 smoothing algorithm to correct the model's probability distribution, the problems of model overconfidence and unclear answer boundaries in multi-hop question answering frameworks are solved, thereby improving the accuracy of retrieval and answer extraction.
Patent Information
- Application Number
- CN202211232474.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-10
- Publication Date
- 2026-02-03
- Estimated Expiration
- 2042-10-10
AI Technical Summary
Existing multi-hop question answering frameworks suffer from overconfidence, leading to high error rates and unclear answer boundaries, resulting in accumulated errors. Existing methods have failed to effectively address this issue.
A label-based smoothing multi-hop answer framework is adopted, which selects the optimal article pairing by combining coarse and fine screening, and uses a pre-trained language model to answer the questions. The F1 smoothing algorithm is used to correct the model's predicted probability distribution.
It significantly improved the model's retrieval accuracy and answer fragment extraction accuracy, solved the problem of model overconfidence, and enhanced the overall performance of the multi-hop question answering framework.
Smart Images
Figure CN117909441B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to a multi-hop question answering framework based on label smoothing and belongs to the technical field of natural language understanding. BACKGROUND
[0002] Modern society is entering an AI era, and large Internet companies have developed large-scale and trained models to enable machines to understand natural language. One important application scenario is to examine the reading comprehension ability of machines, which requires machines to retrieve relevant documents to answer corresponding questions.
[0003] The multi-hop question answering task is different from the simple question answering task. There are multiple "jump points" in the logical chain of answering questions, and each "jump point" has an important influence on the model to obtain a correct answer.
[0004] The existing multi-hop question answering method is generally divided into two stages. In the first stage, a retrieval model is used to select relevant articles from several candidate articles. In the second stage, a machine reading comprehension model is used to reason the final answer from the extracted two articles. In the first stage, the existing method usually considers the interaction relationship between different candidate articles. In the second stage, the existing method uses a graph neural network, and some methods use the powerful ability of a pre-trained model. However, these methods do not consider the correction problem of the model. In the neural network, the model tends to be overly confident in its judgment result, resulting in a higher error rate. The higher error rate has an error accumulation problem for the two-stage model, which makes the error rate of the second stage higher. In addition, in the answer extraction of the second stage, the boundary of the answer is usually not clear. For example, "35" and "35 people" can both be answers to the question "How many people?". However, this phenomenon is not explicitly learned by the model. SUMMARY
[0005] The application is made to solve the above problems, and aims to provide a multi-hop question answering framework based on label smoothing, which can alleviate the problem of overconfidence of the model and the uncertainty of the answer boundary. To this end, the application provides the following technical solutions.
[0006] The application provides a multi-hop question answering framework based on label smoothing, which has the following characteristics: step S1, inputting a question and a list of candidate articles to obtain three articles to be selected; step S2, grouping the three articles to be selected in pairs to form three pairs, and selecting the optimal pair through fine screening; step S3, inputting the two articles of the selected optimal pair and the question into the model to answer the question through a pre-trained language model.
[0007] In the label smoothing based multi-hop question answering framework provided by the application, the function of the experience risk loss in the step S1 can be:
[0008]
[0009] wherein E represents taking expectation for all samples, M represents the number of candidate articles corresponding to a single sample, L i represents the loss function of whether the i-th candidate article is filtered, and is specifically represented as:
[0010]
[0011] wherein, represents whether the i-th article is one of the correct candidate articles, and is 1 if yes, and is 0 if no, represents the probability that the model predicts that the i-th article is one of the correct candidate articles, and log represents a natural logarithm, wherein the can be smoothed by a label smoothing method, that is, is represented as 0.9 when the i-th article is one of the correct candidate articles, and is 0.1 if not.
[0012] In the label smoothing based multi-hop question answering framework provided by the application, the function of the experience risk loss in the step S1 can be:
[0013]
[0014] wherein w i represents the weight of the loss function of the i-th candidate article in a single sample, and is 2 if the article contains the final answer, and the remaining weights are 1.
[0015] In the label smoothing based multi-hop question answering framework provided by the application, the step S2 can comprise the following sub-steps: step S2-1, constructing candidate article pairs, pairing the top three candidate articles selected in the step S1 two by two to generate three article pairs; and step S2-2, inputting the question and the article pairs into the model to select the best pair, wherein the function of the experience risk loss of the step S2 is:
[0016]
[0017] wherein C i represents the i-th article pair in a certain sample, represents whether the i-th article pair is the optimal pair, and is 1 if yes, and is 0 if no, denotes the probability that the i-th pair of articles is the optimal pair predicted by the model. The smoothing is performed by a label smoothing method, so that it is 0.9 when the pair is the optimal pair, and 0.05 otherwise.
[0018] In the label smoothing-based multi-hop question answering framework provided by the application, the step S1 and the step S2 can be simultaneously trained and optimized, and the final goal is:
[0019] L retrieval = φ1L coarse + φ2L fine
[0020] In the formula, φ1 and φ2 are fixed coefficients, φ1 is 2, and φ2 is 1.
[0021] In the label smoothing-based multi-hop question answering framework provided by the application, the step S3 can further include the following sub-steps: step S3-1, concatenating and serializing the input two articles; step S3-2, inputting the processed sequence into a pre-trained model for feature extraction; step S3-3, extracting the start position and end position of the answer by using a reading comprehension model; and step S3-4, outputting the answer restored from the original text according to the obtained start position and end position,
[0022] In the label smoothing-based multi-hop question answering framework provided by the application, the step S3 can further include the following sub-steps: step S3-1, concatenating and serializing the input two articles; step S3-2, inputting the processed sequence into a pre-trained model for feature extraction; step S3-3, extracting the start position and end position of the answer by using a reading comprehension model; and step S3-4, outputting the answer restored from the original text according to the obtained start position and end position,
[0023]
[0024]
[0025]
[0026] In the formula, φ1 and φ2 are fixed coefficients, φ1 is 2, and φ2 is 1. denotes the probability distribution predicted by the start position model, denotes the probability distribution predicted by the end position model, denotes the distribution of the correct start position, that is, only the correct start position is 1, and the remaining positions are 0, denotes the distribution of the correct end position, that is, only the correct end position is 1, and the remaining positions are 0.
[0027] In the label smoothing-based multi-hop question answering framework provided by the application, the step S3 can further include the following sub-steps: step S3-1, concatenating and serializing the input two articles; step S3-2, inputting the processed sequence into a pre-trained model for feature extraction; step S3-3, extracting the start position and end position of the answer by using a reading comprehension model; and step S3-4, outputting the answer restored from the original text according to the obtained start position and end position, and The correction formula is:
[0028]
[0029]
[0030] In the formula, q s (i) represents the corrected probability distribution of the starting position, q e (i) represents the corrected probability distribution of the ending position, F1 represents the F1 score, and a gold This indicates the correct answer.
[0031] The role and effect of invention
[0032] According to the label-smoothing multi-hop answering framework of this invention, three candidate articles are first obtained through coarse screening based on the input question and the candidate article list. Then, these three candidate articles are paired to form three pairs. The optimal pair is then selected through fine screening. This process from coarse screening to fine selection significantly improves the model's retrieval accuracy. Finally, the two selected optimal articles and the question are input into the model, and a pre-trained language model answers the question.
[0033] The label-smoothing multi-hop question answering framework of this invention has achieved excellent performance on well-known reading comprehension datasets. It is simple in design and has a wide range of application scenarios, solving the problem that the performance of existing multi-hop question answering frameworks needs to be improved. Attached Figure Description
[0034] Figure 1 This is a schematic diagram of the overall framework of multiple problem frames based on label smoothing in an embodiment of the present invention;
[0035] Figure 2 This is an architectural flowchart of a multi-problem framework based on label smoothing in an embodiment of the present invention; and
[0036] Figure 3 This is a comparison diagram between the F1 smoothing algorithm proposed in the embodiments of the present invention and the traditional algorithm. Detailed Implementation
[0037] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the multi-problem framework of this invention based on label smoothing.
[0038] <Example>
[0039] Figure 1 This is a schematic diagram of the overall framework of the label-smooth multi-hop answering question framework in an embodiment of the present invention.
[0040] like Figure 1As shown, the label-smoothing multi-hop answering framework consists of three stages. The first stage is to coarsely screen the input question and the candidate article list to obtain three candidate articles. The second stage is to form three pairs by grouping the three candidate articles into pairs and select the optimal pair through fine screening. The third stage is to use the selected two optimal articles and the question to answer the question through a pre-trained language model.
[0041] like Figure 2 As shown, the multi-question framework based on label smoothing includes the following steps:
[0042] Step S1: Input the question and the list of candidate articles to get 3 shortlisted articles.
[0043] To roughly filter for candidate articles, this embodiment constructs an empirical risk loss function using known correct candidate articles, the expression of which is shown below:
[0044]
[0045] In the formula, w i Let represent the weight of the loss function for the i-th candidate article in a single sample. If the article contains the final answer, the weight is 2; otherwise, the weight is 1. E represents the expectation over all samples, M represents the number of candidate articles for a single sample, and L... i The loss function representing whether to screen the i-th candidate article is specifically expressed as:
[0046]
[0047] in, This indicates whether the i-th article is one of the correct candidate articles. Let represent the probability that the model predicts the i-th article as one of the correct candidate articles, and log represents the natural logarithm. This can be achieved through label smoothing techniques. The smoothing process is applied, meaning that the value is 0.9 when the i-th article is one of the correct candidate articles, and 0.1 otherwise.
[0048] Step S2 involves grouping the three candidate articles into three pairs, and then selecting the optimal pair through a fine-tuning process.
[0049] In this embodiment, candidate article pairings are first constructed by pairing the top three candidate articles selected in step S1 to generate three sets of article pairings. Then, the question and article pairings are simultaneously input into the model to select the best pairing. The empirical risk loss function in step S2 is:
[0050]
[0051] In the formula, Ci This represents the pairing of the i-th group of articles in a certain sample. This indicates whether the i-th article pairing is the optimal pairing; if it is, it is 1, otherwise it is 0. This represents the probability that the model predicts the i-th article pairing to be the optimal pairing. For The label smoothing method is used to smooth the data so that the value is 0.9 when the pairing is the optimal pairing and 0.05 otherwise.
[0052] In step S2, the model training process is as follows:
[0053] Step S2-1: Construct candidate article pairings by pairing the three candidate articles with the highest probabilities selected in step S1 into three groups of article pairings.
[0054] Step S2-2: Pair the question with the article and input the data into the model to select the best pairing.
[0055] In this embodiment, steps S1 and S2 are trained and optimized simultaneously, with the final goal being:
[0056] L retrieval =φ1L coarse +φ2L fine
[0057] In the formula, Φ1 and Φ2 are fixed coefficients, with Φ1 being 2 and Φ2 being 1. coarse Let L represent the loss function for step S1. fine Let S2 be the loss function.
[0058] Step S3: Input the two best selected articles and the question into the model, and answer the question through the pre-trained language model.
[0059]
[0060] In the formula, This represents the probability distribution predicted by the model at the starting position. This represents the probability distribution predicted by the model at the end position. This represents the distribution of the correct starting positions, where only the correct starting position is 1, and all other positions are 0. This represents the distribution of correct ending positions, where only the correct ending position is 1, and the rest are 0. m f represents the semantic representation of the specific words (Token level) that constitute the sample. start and f end These are mapping functions for the start and end positions of the answer, respectively. Specifically:
[0061] The specific words (Token level) that constitute the sample with the highest probability at the start and end positions are used as the start and end points of the answer, and these are used as the output.
[0062] Figure 3 This is a comparison of the effects of the present invention and existing multi-hop question answering methods in the embodiments of the present invention.
[0063] In this embodiment, the label-smoothing multi-hop answering method of this invention achieves the best performance across all metrics compared to previous methods on the HotpotQA multi-hop answering question dataset. Here, Ans represents the answer, Sup represents the basis, Joint represents the sum of the answer and basis metrics, EM represents the exact match accuracy, and F1 represents the F1 score.
[0064] The role and effect of the embodiments
[0065] Based on the label-smoothing multi-hop question-answering framework provided in this example, the model first coarsely filters out three candidate articles from the input question and the candidate article list. Then, these three candidate articles are paired to form three pairs. A further fine-tuning process selects the optimal pair. This process, from coarse to fine selection, significantly improves the model's retrieval accuracy. Finally, the two selected optimal articles and the question are input into the model, which answers the question using a pre-trained language model. The F1 smoothing algorithm is employed to significantly alleviate the model's overconfidence and improve the accuracy of answer fragment extraction.
[0066] In the embodiments, the present invention achieved the best results in all indicators.
[0067] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.
Claims
1. A label-smoothing-based multi-hop answering framework to alleviate the problems of model overconfidence and uncertain answer boundaries, characterized in that, Includes the following steps: Step S1: Input the question and the list of candidate articles to obtain 3 shortlisted articles; Step S2: The three candidate articles are grouped into three pairs, and the best pair of articles is selected through a fine screening process. as well as Step S3: Input the two selected optimally paired articles and the question into the model, and answer the question using a pre-trained language model. In step S1, an empirical risk loss function is constructed using known correct candidate articles, and its expression is: In the formula, w i Let represent the weight of the loss function for the i-th candidate article in a single sample. If the article contains the final answer, the weight is 2; otherwise, the weight is 1. E represents the expectation over all samples, M represents the number of candidate articles corresponding to a single sample, and L... i The loss function representing whether to screen the i-th candidate article is specifically expressed as: In the formula, This indicates whether the i-th article is one of the correct candidate articles; a value of 1 indicates it is, and a value of 0 indicates it is not. Let represent the probability that the model predicts the i-th article as one of the correct candidate articles, and let log represent the natural logarithm. This can be achieved through label smoothing. The smoothing process is applied, meaning that the value is 0.9 when the i-th article is one of the correct candidate articles, and 0.1 otherwise. Step S3 includes the following sub-steps: Step S3-1: Concatenate the two input articles and then serialize them; Step S3-2: Input the processed sequence into the pre-trained model for feature extraction; Step S3-3: Use a reading comprehension model to extract the start and end positions of the answer; Steps S3-4: Based on the obtained start and end positions, reconstruct the answer from the original text and output it. The empirical risk loss function used in step S3 during training is as follows: In the formula, This represents the probability distribution predicted by the model at the starting position. This represents the probability distribution predicted by the model at the end position. This represents the distribution of the correct starting positions, where only the correct starting position is 1, and all other positions are 0. This represents the distribution of correct ending positions, where only the correct ending position is 1, and all other positions are 0. In step S3, to better avoid the problem of model overconfidence, the F1Smoothing method is used for correction. and The corrected formula is: Formula q s In (i), q represents the corrected probability distribution of the starting position. e (i) represents the corrected probability distribution of the ending position, F1 represents the F1 score, and a gold This indicates the correct answer.
2. The label-smoothing multi-hop answering framework according to claim 1, characterized in that: in, Step S2 includes the following sub-steps: Step S2-1: Construct candidate article pairings by pairing the top three candidate articles selected in step S1 with each other to generate three sets of article pairings. Step S2-2: Pair the question with the article and input the data into the model to select the best pairing. The function of the empirical risk loss in step S2 is: In the formula, C i This represents the pairing of the i-th group of articles in a certain sample. This indicates whether the i-th article pairing is the optimal pairing; a value of 1 indicates it is, and a value of 0 indicates it is not. This represents the probability that the model predicts the i-th article pairing as the optimal pairing. The label smoothing method is used to smooth the data so that the value is 0.9 when the pairing is the optimal pairing and 0.05 otherwise.
3. The label-smoothing multi-hop answering framework according to claim 2, characterized in that: in, The simultaneous training and optimization in steps S1 and S2 have the following final goal: L retrieval =φ1L coarse +φ2L fine In the formula, Φ1 and Φ2 are both fixed coefficients, with Φ1 being 2 and Φ2 being 1.