An advanced training-based few-shot extractive question answering method

Abstract

The application relates to the technical field of extractive question answering tasks, and particularly relates to a few-shot extractive question answering method based on advanced training, which comprises the following steps: constructing pseudo question and answer pair data and question and answer pair data with unified data formats; hiding entities or noun phrases at the end of questions in the pseudo question and answer pair data; obtaining a pre-training language model; performing advanced training on the pre-training language model through the pseudo question and answer pair data to obtain an advanced training model; fine-tuning the advanced training model to obtain a final language model; and inputting to-be-predicted question texts and given context texts into the final language model to predict corresponding answers. The application can effectively improve the task differences and input data format differences existing in the pre-training stage and the fine-tuning stage of the pre-training language model, thereby improving the effectiveness and practicality of the few-shot extractive question answering, and providing a new solution for the few-shot extractive question answering task.

Description

Technical Field

[0001] This invention relates to the field of extraction-based question answering tasks, and more specifically to a few-sample extraction-based question answering method based on advanced training. Background Technology

[0002] Extractive question answering (EQA) is a fundamental yet crucial task in natural language processing. It can serve as a foundation for multi-turn dialogues and open-domain question answering, or it can exist independently as a question-answering task. The EQA task aims to predict the start and end points of the answer to a question given a question and its related context, using appropriate machine learning or deep learning methods. Due to its unique task characteristics, EQA can be widely applied in fields such as intelligent question answering systems, search engines, and intelligent assistants.

[0003] To improve the performance of extractive question answering and enable its rapid and accurate application in real-world question answering scenarios, researchers have proposed various methods and models for this task, including BiDAF and R-Net. Existing technologies also include the pre-trained language model BERT, which significantly improves the performance of natural language processing tasks by introducing mask prediction and next-sentence prediction as pre-training tasks and fine-tuning them in downstream tasks. This has also driven the development of natural language processing tasks towards a "pre-training, fine-tuning" paradigm. Based on this type of pre-trained language model, the performance of extractive question answering tasks has continuously improved, achieving excellent results on some datasets, even surpassing human question answering levels. However, the performance improvements achieved by pre-trained language models in extractive question answering tasks rely on large amounts of labeled data. In real-world applications, labeling large amounts of data is time-consuming and labor-intensive, and many vertical domains require expert knowledge to label high-quality training data.

[0004] Therefore, to improve the performance of existing models in extractive question answering with limited labeled data and further expand the application scenarios and fields of extractive question answering tasks, research on few-shot extractive question answering tasks has emerged. Compared to full-sample extractive question answering tasks, in few-shot extractive question answering, due to the scarcity of training data, the differences between existing pre-trained language models in the training task and downstream extractive question answering cannot be effectively eliminated. Therefore, how to improve the task differences and input data format differences between pre-trained language models in the pre-training and fine-tuning stages is an urgent technical problem to be solved. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the technical problem this invention aims to solve is: how to provide a few-shot extraction question answering method based on advanced training, which can effectively improve the differences in task and input data format between the pre-training and fine-tuning stages of pre-trained language models, thereby improving the effectiveness and practicality of few-shot extraction question answering, and providing a new solution for few-shot extraction question answering tasks.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A few-shot extraction question answering method based on advanced training includes:

[0008] S1: Construct pseudo-question-answer pairs and question-answer pairs with a unified data format; in the pseudo-question-answer pairs, entities or noun phrases at the end of the questions are masked.

[0009] S2: Obtain the pre-trained language model;

[0010] S3: Use pseudo-question answering to perform advanced training on the pre-trained language model to obtain an advanced trained model;

[0011] S4: Fine-tune the advanced training model to obtain the final language model;

[0012] S5: Input the question text to be predicted and the given context text into the final language model to predict the corresponding answer.

[0013] Preferably, pseudo-question-answer pairs are constructed through the following steps:

[0014] S101: Collect unlabeled data and perform data cleaning and preprocessing to obtain several long text segments P;

[0015] S102: Use natural language processing tools to segment each long text paragraph P into sentences, resulting in several sentences S; where P = (S1, S2, ..., S...). t ), where t represents the number of sentences in the long text paragraph P;

[0016] S103: For long text paragraphs P: Select sentences S in sequence i As a question And select S i+1 ,S i+2 ,…,S t Contextual text related to question Q

[0017] S104: Locating entities that exist at the end of problem Q or noun phrase The entity or noun phrase obtained by location is masked by the token[QUESTION], and then combined with the corresponding context text C to construct a set of pseudo-question-answer pairs.

[0018] In the formula: T represents a set of pseudo-question-answer pairs; [CLS] and [SEP] represent the special tokens required for the input data of the pre-trained language model.

[0019] Preferably, the entities or noun phrases obtained by locating must meet the following requirements: or

[0020] Preferably, question-answer pair data is constructed through the following steps:

[0021] S211: Collect unlabeled data and perform data cleaning and preprocessing to obtain several long text segments P;

[0022] S212: Use natural language processing tools to segment each long text paragraph P into sentences, resulting in several sentences S; where P = (S1, S2, ..., S...). t ), where t represents the number of sentences in the long text paragraph P;

[0023] S213: For long text paragraphs P: Select sentences S in sequence i As a question And select S i+1 ,S i+2 ,…,S t Contextual text related to question Q

[0024] S214: Add a token [QUESTION] after question Q, and combine it with the corresponding context text C to construct a set of question-answer pairs.

[0025] In the formula: T′ represents a set of question-answer pairs; [CLS] and [SEP] represent the special tokens required for the input data of the pre-trained language model.

[0026] Preferably, data cleaning and preprocessing includes: removing non-text characters from unlabeled data, merging multiple newline characters in unlabeled data, and segmenting unlabeled data using newline characters.

[0027] Preferably, the pre-trained language model is a Splinter model, an ALBert model, a SpanBERT model, or a Roberta model.

[0028] Preferably, the pre-trained language model includes a BERT encoder and a parameter matrix M. S and ME The question-and-answer segment selection module first inputs the question and the given text into the BERT encoder to obtain the corresponding hidden layer word vectors; then, the hidden layer word vectors are input into the question-and-answer segment selection module and normalized and processed to obtain the probabilities of the start and end positions of the predicted answer.

[0029] The formula is described as follows:

[0030] h = w BERT (T);

[0031]

[0032]

[0033] In the formula: p(s=i|T) represents the probability that the i-th token is the starting position of the answer; p(e=i|T) represents the probability that the i-th token is the ending position of the answer; w BERT Represents the weights of the BERT encoder in the pre-trained language model; M represents the hidden layer word vectors obtained after encoding by the BERT encoder; S and M E represents the parameter matrix of the question-and-answer segment selection module; T represents a set of pseudo-question-and-answer pairs; n represents the total number of tokens input to the model; i represents the index of the token.

[0034] Preferably, the loss value during advanced training is calculated using a loss function, and the model parameters are updated using a backpropagation algorithm to gradually reduce the magnitude of the loss value, thereby achieving advanced training of the pre-trained language model.

[0035] The loss value for advanced training is calculated using the following loss function:

[0036] loss = CrossEntropyLoss(start) g ,p(s|T))+CrossEntropyLoss(end g ,p(e|T));

[0037] In the formula: loss represents the loss value during advanced model training; CrossEntropyLoss represents the cross-entropy loss function; start g Indicates the starting position of the true answer; end g p(s|T) represents the ending position of the true answer; p(s|T) represents the probability that the position is the beginning position of the answer among all positions; p(e|T) represents the probability that the position is the ending position of the answer among all positions.

[0038] Preferably, when fine-tuning the advanced training model: first, retain the parameters of the BERT encoder of the advanced training model; then, reinitialize the parameters of the question-answering segment selection module of the advanced training model; and finally, train the advanced training model using question-answering pair data.

[0039] Preferably, the final language model processing procedure is described by the following formula:

[0040] start = argmax(p(s|T));

[0041] end = argmax(p(e|T));

[0042] Answer = C′[start:end];

[0043] In the formula: start and end represent the start and end positions of the predicted answer; p(s|T) represents the probability of being the start position of the answer among all positions; p(e|T) represents the probability of being the end position of the answer among all positions; the argmax function means taking the index corresponding to the maximum value in the input vector, that is, extracting the index of the largest value between p(s|T) and p(e|T) as the start and end positions of the predicted answer; Answer represents the predicted answer; C′ represents the given context text.

[0044] Compared with existing technologies, the few-shot extraction question-answering method based on advanced training in this invention has the following advantages:

[0045] In practical applications, pre-trained language models exhibit significant task and input data format differences during the pre-training and fine-tuning stages. However, limited input data prevents the model from overcoming these task and data differences, hindering the performance of extractive question answering (EQA) to meet practical application standards. To address this, this invention, building upon a pre-trained language model suitable for natural language processing, utilizes constructed pseudo-question-answer pairs for advanced training. This advanced training effectively mitigates the task and input data format differences between the pre-training and fine-tuning stages, achieving adaptive EQA and enabling performance to meet practical application standards. Furthermore, the pseudo-question-answering mechanism masks entities or noun phrases at the end of the question, guiding the pre-trained language model to identify text fragments identical to these masked entities or phrases in other parts of the sentence. This aligns the advanced training objective with the EQA task, thereby improving the effectiveness and practicality of few-shot EQA.

[0046] In practical applications, differences in the input data structure can introduce task discrepancies between the pre-training and fine-tuning stages. This forces the model to require more input data to overcome these data and task differences during fine-tuning, resulting in a significant performance drop for the same amount of input data. To address this, this invention performs fine-tuning based on advanced training. Furthermore, the pseudo-question-answer pairs and question-answer pairs constructed during both advanced training and fine-tuning are formatted uniformly. This unified data format effectively eliminates the differences between the pre-training and fine-tuning stages, thereby improving the performance of the model's extractive question-answering capabilities and further enhancing the effectiveness of few-shot extractive question answering. Attached Figure Description

[0047] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0048] Figure 1 The flowchart shows the logic of a few-shot extraction question answering method based on advanced training.

[0049] Figure 2 This is the overall flowchart of a few-shot extraction question answering method based on advanced training. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship commonly used when the product is in use. They are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. In addition, the terms "horizontal," "vertical," etc., do not mean that the component is required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0052] The following detailed explanation illustrates the specific implementation methods:

[0053] Example:

[0054] This embodiment discloses a few-shot extraction question-answering method based on advanced training.

[0055] like Figure 1 and Figure 2 As shown, the few-shot extraction question answering method based on advanced training includes:

[0056] S1: Construct pseudo-question-answer pairs and question-answer pairs with a unified data format; in the pseudo-question-answer pairs, entities or noun phrases at the end of the questions are masked.

[0057] S2: Obtain the pre-trained language model;

[0058] S3: Post-training the pre-trained language model using pseudo-question answering data to obtain an advanced trained model;

[0059] S4: Fine-tune the advanced training model to obtain the final language model;

[0060] S5: Input the question text to be predicted and the given context text into the final language model to predict the corresponding answer.

[0061] In practical applications, pre-trained language models exhibit significant task and input data format differences during the pre-training and fine-tuning stages. However, limited input data prevents the model from overcoming these task and data differences, hindering the performance of extractive question answering (EQA) to meet practical application standards. To address this, this invention, building upon a pre-trained language model suitable for natural language processing, utilizes constructed pseudo-question-answer pairs for advanced training. This advanced training effectively mitigates the task and input data format differences between the pre-training and fine-tuning stages, achieving adaptive EQA and enabling performance to meet practical application standards. Furthermore, the pseudo-question-answering mechanism masks entities or noun phrases at the end of the question, guiding the pre-trained language model to identify text fragments identical to these masked entities or phrases in other parts of the sentence. This aligns the advanced training objective with the EQA task, thereby improving the effectiveness and practicality of few-shot EQA.

[0062] In practical applications, differences in the input data structure can introduce task discrepancies between the pre-training and fine-tuning stages. This forces the model to require more input data to overcome these data and task differences during fine-tuning, resulting in a significant performance drop for the same amount of input data. To address this, this invention performs fine-tuning based on advanced training. Furthermore, the pseudo-question-answer pairs and question-answer pairs constructed during both advanced training and fine-tuning are formatted uniformly. This unified data format effectively eliminates the differences between the pre-training and fine-tuning stages, thereby improving the performance of the model's extractive question-answering capabilities and further enhancing the effectiveness of few-shot extractive question answering.

[0063] In the specific implementation process, pseudo-question-answer pairs of data are constructed through the following steps:

[0064] S101: Collect a large amount of unlabeled data and perform data cleaning and preprocessing to obtain several long text paragraphs P;

[0065] In this embodiment, data cleaning and preprocessing include deleting non-text characters (HTML tags, non-text Unicode characters, etc.) from unlabeled data, merging multiple line breaks in unlabeled data, and segmenting unlabeled data using line breaks.

[0066] To ensure that the subsequently generated pseudo-data has sufficient context and accurately locates the masked entities or noun phrases, we segment each long text segment P into words, transforming each long text segment P into a combination of multiple tokens, i.e. x p L represents a single token after segmentation of P. p Let P be the number of tokens in paragraph P. Then, we delete text paragraphs with fewer than 300 tokens, keeping only long text paragraphs with rich contextual relationships as input data for subsequent data construction processes, thus obtaining the final long text paragraph P.

[0067] S102: Use the natural language processing tool spaCy to segment each long text segment P into sentences, resulting in several sentences S; where P = (S1, S2, ..., S...). t ), where t represents the number of sentences in the long text paragraph P;

[0068] S103: For long text paragraphs P: Select sentences S in sequence i As a question And select S i+1 ,S i+2 ,…,S t Contextual text related to question Q

[0069] In this embodiment, to ensure that the subsequently generated pseudo-question-answer pairs have sufficient contextual relationships, we still delete the contextual text C with less than 300 tokens.

[0070] S104: (Using spaCy tool) Locate entities that exist at the end of problem Q. or noun phrase Then, by masking the entities or noun phrases obtained through location using token[QUESTION], and combining this with the corresponding context text C, a set of pseudo-question-answer pairs is constructed:

[0071]

[0072] In the formula: T represents a set of pseudo-question-answer pairs; [CLS] and [SEP] represent special tokens required for the input data of the pre-trained language model. Specifically, the final number of tokens in the pseudo-question-answer pair data T is 384. If the total number of tokens in Q′ and the context text C is greater than 381 (after removing 1 [CLS] token and 2 [SEP] tokens), then the context text C is truncated to meet the length requirement. If the total number of tokens in Q′ and the context text C is less than 381, then a [SEP] token is added after the context text C to meet the length requirement.

[0073] In this embodiment, the located entity or noun phrase must meet the following requirements: or

[0074] After the data is constructed, we search for text fragments in the context text C of the question-answer pair that are identical to the masked entity E or noun phrase N. If there exists a text fragment in C with start and end tokens respectively... If a text fragment is found in the context text C, then the start and end positions of that fragment are taken as the start and end positions of the true answer in the pseudo-question-answer pair data T, and the question-answer pair is marked as an answerable question. If the fragment cannot be found in the context text C, then the start and end positions of the true answer are both set to 0, indicating that the question-answer pair data is an unanswerable question. The final construction example is shown in Table 1.

[0075] Table 1 Examples of pseudo-question answer pairs

[0076]

[0077]

[0078] This invention collects a large amount of unlabeled long text and uses existing natural language processing tools to segment the unlabeled long text into sentences, identifying the entities and noun phrases contained therein. Then, entities or noun phrases that meet the conditions are masked, and pseudo-question-answer pairs (input data) are constructed to train the model. This allows the pre-trained language model to better find text segments in other positions in the sentence that are the same as the masked entities or noun phrases. In this way, the advanced training objective is aligned with the extraction question-answering task, thereby further improving the effectiveness of few-sample extraction question answering.

[0079] In the specific implementation process, the question-answer pair data is constructed through the following steps:

[0080] S211: Collect a large amount of unlabeled data and perform data cleaning and preprocessing to obtain several long text paragraphs P;

[0081] In this embodiment, data cleaning and preprocessing include deleting non-text characters (HTML tags, non-text Unicode characters, etc.) from unlabeled data, merging multiple line breaks in unlabeled data, and segmenting unlabeled data using line breaks.

[0082] To ensure that the subsequently generated pseudo-data has sufficient context and accurately locates the masked entities or noun phrases, we segment each long text segment P into a combination of multiple tokens, i.e., P = x p L represents a single token after segmentation of P. p Let P be the number of tokens in paragraph P. Then, we delete text paragraphs with fewer than 300 tokens, keeping only long text paragraphs with rich contextual relationships as input data for subsequent data construction processes, thus obtaining the final long text paragraph P.

[0083] S212: Use the natural language processing tool spaCy to segment each long text segment P into sentences, resulting in several sentences S; where P = (S1, S2, ..., S...). t ), where t represents the number of sentences in the long text paragraph P;

[0084] S213: For long text paragraphs P: Select sentences S in sequence i As a question And select S i+1 ,S i+2 ,…,S t Contextual text related to question Q

[0085] In this embodiment, to ensure that the subsequently generated pseudo-question-answer pairs have sufficient contextual relationships, we still delete the contextual text C with less than 300 tokens.

[0086] S214: Add a token [QUESTION] after question Q, and combine it with the corresponding context text C to construct a set of question-answer pairs.

[0087] In the formula: T′ represents a set of question-answer pairs; [CLS] and [SEP] represent the special tokens required for the input data of the pre-trained language model.

[0088] In this invention, the question-answer pair data used for fine-tuning is constructed in the same way as the pseudo-question-answer pair data, but there is no need to mask entities or noun phrases. This makes the format of the input data in the advanced training and fine-tuning stages exactly the same, avoiding the data difference between the pre-training and fine-tuning stages, thereby improving the performance of the model's extractive question answering and further improving the effectiveness of few-sample extractive question answering.

[0089] In the specific implementation process, the pre-trained language model selected is the Splinter model, ALBert model, SpanBERT model, or Roberta model.

[0090] In this embodiment, the optimal pre-trained language model is the Splinter model. In other preferred embodiments, Splinter can be replaced with any other pre-trained language model, including ALBert, SpanBERT, Roberta, etc. The final performance of the few-shot question answering task will decrease slightly, but the impact is not significant.

[0091] Specifically, the pre-trained language model includes the BERT encoder and a parameter matrix M. S and M E The question-and-answer segment selection module (QASS) is used. First, the question and given text are input into the BERT encoder to obtain the corresponding hidden layer word vectors. Then, the hidden layer word vectors are input into the question-and-answer segment selection module and normalized to obtain the probabilities of the predicted start and end positions of the answer.

[0092] The formula is described as follows:

[0093] h = w BERT (T);

[0094]

[0095]

[0096] In the formula: p(s=i|T) represents the probability that the i-th token is the starting position of the answer; p(e=i|T) represents the probability that the i-th token is the ending position of the answer; w BERT Represents the weights of the BERT encoder in the pre-trained language model; M represents the hidden layer word vectors obtained after encoding by the BERT encoder; S and M E represents the parameter matrix of the question-and-answer segment selection module; T represents a set of pseudo-question-and-answer pairs; n represents the total number of tokens input to the model; j represents the loop variable; i represents the index of the token.

[0097] In the specific implementation process, the loss value during advanced training is calculated through the loss function, and the model parameters are updated using the backpropagation algorithm to gradually reduce the magnitude of the loss value in order to achieve advanced training of the pre-trained language model.

[0098] The loss value for advanced training is calculated using the following loss function:

[0099] loss = CrossEntropyLoss(start) g ,p(s|T))+CrossEntropyLoss(end g ,p(e|T));

[0100] In the formula: `loss` represents the loss value during advanced model training, calculated using the cross-entropy loss function. Backpropagation is used to update model parameters, gradually reducing the value of the loss to achieve the desired training effect; `CrossEntropyLoss` represents the cross-entropy loss function; `start`... g Indicates the starting position of the true answer; end g p(s|T) represents the ending position of the true answer; p(s|T) represents the probability that the position is the beginning position of the answer among all positions; p(e|T) represents the probability that the position is the ending position of the answer among all positions.

[0101] In the specific implementation process, when fine-tuning the advanced training model: first, retain the parameters of the BERT encoder of the advanced training model; then, reinitialize the parameters of the question-and-answer segment selection module of the advanced training model; finally, train the advanced training model using question-and-answer pair data.

[0102] In this embodiment, during the fine-tuning process, the loss is calculated using the same method as during advanced training, i.e., by the formula loss = CrossEntropyLoss(start) g ,p(s|T))+CrossEntropyLoss(end g The loss is calculated using p(e|T) to optimize the model.

[0103] In other preferred embodiments, the question-and-answer segment selection module can also use the original weights of the advanced training model. There is no significant performance change on datasets where the distribution of most advanced training data and fine-tuning data is similar, and the performance only decreases when the distribution difference is large.

[0104] In practice, after fine-tuning the model, the model, after advanced training and fine-tuning, has a strong ability to extract questions and answer them, and can accurately extract text fragments from the context text C as the predicted answer to question Q.

[0105] The final language model processing procedure is described by the following formula:

[0106] start = argmax(p(s|T));

[0107] end = argmax(p(e|T));

[0108] Answer = C′[start:end];

[0109] In the formula: start and end represent the start and end positions of the predicted answer; p(s|T) represents the probability of being the start position of the answer among all positions; p(e|T) represents the probability of being the end position of the answer among all positions; the argmax function means taking the index corresponding to the maximum value in the input vector, that is, extracting the index of the largest value between p(s|T) and p(e|T) as the start and end positions of the predicted answer; Answer represents the predicted answer; C′ represents the given context text.

[0110] In this embodiment, the method proposed in this invention was used to conduct a large number of experiments on four datasets: SQuAD, TriviaQA, NaturalQuestions, and NewsQA. The number of input data was selected as 16, 32, and 64 respectively to simulate a few-shot question answering task scenario. The specific results are shown in Table 2.

[0111] Table 2 Results of Advanced Training Experiment

[0112]

[0113]

[0114] As can be seen from Table 2, there are two key steps in the specific implementation of the method of the present invention:

[0115] The first key step is advanced training. Without advanced training, pre-trained language models will have significant differences in tasks and input data formats during the pre-training and fine-tuning stages. The limited input data makes it impossible for the model to overcome the differences in tasks and data, and the performance of extractive question answering cannot meet the standards for practical applications.

[0116] The second crucial step is standardizing the input data format. During both the advanced training and fine-tuning phases, the input data must be constructed in the same format. This is a key step in eliminating discrepancies between the pre-training and fine-tuning stages. If the input data format differs, it will introduce task differences between the pre-training and fine-tuning stages. This means that during fine-tuning, the model will require more input data to overcome these data and task differences, and with the same amount of input data, model performance will be significantly reduced.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A few-shot question answering method based on advanced training, characterized in that, include: S1: Construct pseudo-question-answer pairs and question-answer pairs with a unified data format; Pseudo-question answering involves masking entities or noun phrases that appear at the end of the question in the data; S2: Obtain the pre-trained language model; The pre-trained language model includes the BERT encoder and a matrix containing two parameter matrices. and The question-and-answer segment selection module first inputs the question and the given text into the BERT encoder to obtain the corresponding hidden layer word vectors; then, the hidden layer word vectors are input into the question-and-answer segment selection module and normalized and processed to obtain the probabilities of the start and end positions of the predicted answer. The formula is described as follows: ； ； ； In the formula: Indicates the first The probability of each token being the starting position of the answer; Indicates the first The probability of each token being the ending position of the answer; Represents the weights of the BERT encoder in the pre-trained language model; This represents the hidden layer word vectors obtained after encoding by the BERT encoder; and This represents the parameter matrix of the question-and-answer segment selection module; This represents a set of pseudo-question-answer pairs; This represents the total number of tokens input into the model; Represents the loop variable; Indicates the index of the token; S3: Use pseudo-question answering to perform advanced training on the pre-trained language model to obtain an advanced trained model; S4: Fine-tune the advanced training model to obtain the final language model; In step S4, when fine-tuning the advanced training model: first, retain the parameters of the BERT encoder of the advanced training model; then, reinitialize the parameters of the question-answering segment selection module of the advanced training model; finally, train the advanced training model using question-answering pair data. S5: Input the question text to be predicted and the given context text into the final language model to predict the corresponding answer.

2. The few-shot extraction question answering method based on advanced training as described in claim 1, characterized in that, In step S1, pseudo-question-answer pair data is constructed through the following steps: S101: Collect unlabeled data and perform data cleaning and preprocessing to obtain several long text paragraphs. ; S102: Use natural language processing tools to process each long text paragraph Segment by clause, resulting in several sentences. ;in , Representing long text paragraphs The number of sentences in the text; S103: For long text paragraphs Select sentences in sequence As a question , , and select As a question Related context text ; S104: Location Problem Entities existing at the end or noun phrase It also uses token[QUESTION] to mask the entities or noun phrases obtained through location, and then combines them with the corresponding context text. A set of pseudo-question-answer pairs was constructed. ; In the formula: [CLS] represents a set of pseudo-question-answer pairs; [CLS] and [SEP] represent special tokens required for the input data of the pre-trained language model.

3. The few-shot extraction question answering method based on advanced training as described in claim 2, characterized in that: In step S104, the located entities or noun phrases must meet the following requirements: or .

4. The few-shot extraction question-answering method based on advanced training as described in claim 1, characterized in that, In step S1, question-answer pair data is constructed through the following steps: S111: Collect unlabeled data and perform data cleaning and preprocessing to obtain several long text paragraphs. ; S112: Use natural language processing tools for each long text paragraph Segment by clause, resulting in several sentences. ;in , Representing long text paragraphs The number of sentences in the text; S113: For long text paragraphs Select sentences in sequence As a question , , and select As a question Related context text ; S114: In the question Then add token[QUESTION], along with the corresponding context text. A set of question-answer pairs data was constructed. ; In the formula: [CLS] represents a set of question-answer pairs; [CLS] and [SEP] represent special tokens required for the input data of the pre-trained language model.

5. The few-shot extraction question answering method based on advanced training as described in claim 2 or 4, characterized in that: Data cleaning and preprocessing include: removing non-text characters from unlabeled data, merging multiple newline characters in unlabeled data, and segmenting unlabeled data based on newline characters.

6. The few-shot extraction question answering method based on advanced training as described in claim 1, characterized in that: In step S2, the pre-trained language model is selected from Splinter, ALBert, SpanBERT, or Roberta models.

7. The few-shot extraction question answering method based on advanced training as described in claim 1, characterized in that, In step S3, the loss value during advanced training is calculated using the loss function, and the model parameters are updated using the backpropagation algorithm to gradually reduce the magnitude of the loss value in order to achieve advanced training of the pre-trained language model. The loss value for advanced training is calculated using the following loss function: ； In the formula: This represents the loss value during advanced training of the model. Represents the cross-entropy loss function; Indicates the starting position of the true answer; Indicates the end position of the true answer; This represents the probability that the position is the starting position of the answer among all possible positions. This represents the probability that the position is the end position of the answer among all possible positions.

8. The few-shot extraction question answering method based on advanced training as described in claim 1, characterized in that: In step S5, the final language model processing procedure is described by the following formula: ； ； ； In the formula: and Indicates the start and end positions of the predicted answer; This represents the probability that the position is the starting position of the answer among all possible positions. This represents the probability that the position is the end position of the answer among all possible positions; The function represents extracting the index corresponding to the maximum value in the input vector, i.e., extracting. and The index with the largest median value is used as the start and end position of the predicted answer; Indicates a predicted answer; Represents the given context text.

Images