Weak semantic association extraction type question answering method based on few-sample prompt learning
By constructing the FSPM4WSAEQA model, using few sample prompts to learn to extract answers in weak semantic correlation scenarios, the problem of insufficient answer extraction ability in weak semantic correlation questions and answers is solved, and efficient correlation discovery and answer extraction in small sample scenarios are achieved.
Patent Information
- Application Number
- CN202411182690.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-03
AI Technical Summary
In weak semantic association scenarios, existing extraction question-and-answer models are difficult to effectively discover the association between question-and-answer pairs, especially in the case of small samples, which leads to insufficient answer extraction capabilities.
Using a method based on few sample prompt learning, the FSPM4WSAEQA model is constructed. By defining contextual text, questions and answer sets, using K+1-Way Few-Shot dataset for model training, fine-tuning the pre-trained language model, obtaining the classification model fNN, and then Extract answers in weak semantic association scenarios.
It significantly improves the model's answer extraction ability in weak semantic correlation scenarios, improves the ability to discover relationships between question and answer pairs, and enhances the performance in small sample scenarios.
Smart Images

Figure CN119066173A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of extractive question answering tasks, and in particular to a weak semantic association extractive question answering method based on few-sample prompt learning. Background Art
[0002] In recent years, as ChatGPT has ignited the chatbot craze, academia and industry have regained their interest in artificial intelligence automatic question answering (QA) systems. Technology giants such as Microsoft, Google, and Baidu have deeply integrated QA with search engines, which not only redefines the boundaries of human-computer interaction, but also profoundly affects the research direction and depth of QA technology. As the key to Turing test to detect machine intelligence, QA has always been an important research direction in the field of natural language processing (NLP). QA tasks can be divided into extractive QA (EQA) and generative QA (GQA) according to different answering methods. Among them, EQA directly extracts key information from context text (Context) through natural language questions, and accurately retrieves and efficiently acquires knowledge in low-value density text big data. EQA not only requires the algorithm to have a strong understanding of the context, but also tests its accuracy in locating key information in a complex text environment. Compared with the "wild and imaginative" answers of GQA, the answers of EQA can be "traced back to the source" and the confidence of the answers can be guaranteed by the "source". It has a wide range of applications in information retrieval, knowledge base construction, intelligent customer service, etc.
[0003] With the introduction of pre-trained language models (PLMs), the performance of EQA tasks has been greatly improved. The continuous refreshing of SOTA (State-Of-The-Art) has made various applications possible and gradually become the focus of research. Current EQA research mainly focuses on model construction, model optimization, and question-answering in specific fields, and many different types of data sets have been adapted for different scenario studies. The weak semantic association EQA task requires the system to have not only text understanding capabilities, but also the ability to reason and judge in the context of weak semantic text. However, the weak semantic association question-answering scenario is still an unresolved challenge. In practical applications, there are still a large number of question-answer pairs with weak semantic relationships, that is, the connection between questions and answers is not obvious. Even if these question-answer pairs are mapped to a high-dimensional semantic space, it is difficult to effectively narrow the distance between them. Therefore, after the models and methods suitable for classic datasets are optimized, it is difficult to obtain satisfactory results on weak semantic association question-answer pairs. Summary of the invention
[0004] In view of this, the present invention provides a weak semantic association extraction question answering method based on few-sample prompt learning, which is used to discover the association between question and answer pairs for the EQA system through prompt learning in weak semantic association scenarios in a few-sample scenario, thereby improving the model's answer extraction ability in weak semantic association scenarios.
[0005] In a first aspect, the present invention provides a weak semantic association extraction question answering method based on few-sample prompt learning, the method comprising:
[0006] Step 1: Define the questions of extractive question answering, including the context text set Context, the question set Query and the answer set Answer;
[0007] Step 2: Build the model FSPM4WSAEQA based on few-sample prompt learning and train the model to obtain the classification model f NN ;
[0008] Step 3: Apply the obtained classification model f NN , extract answers from a given context text collection based on the question.
[0009] Optionally, step 1 includes:
[0010] Define the context text set Context, the question set Query and the answer set Answer, where Context = {clause1, clause2,…, clause N} represents a context text set consisting of several clauses, and N is a positive integer; Query = {query1, query2, ..., query K} represents a set of K questions raised for the context text set Context, where K is a positive integer; Answer = {answer1, answer2, …, answer K} represents K answer sets extracted from the context text set Context for K questions, and each answer subset answer i ={clause i1 ,clause i2 ,…,clause iM} are all subsets of the context text set Context, i∈[1,K], that is,
[0011] Optionally, the step 2 includes:
[0012] Each type of question query in the question set Query i Mapped to category label label i; and create category labels for questions not included in the question set Query K+1 , all categories form a label set Label = {label1, label2, ..., label K ,label K+1}; Use prompt learning to define the template Template = template ([clause], [label]), and use the few-sample question-answer data to instantiate the template, and then fine-tune the pre-trained language model PLMs to obtain the classification model f NN .
[0013] Optionally include:
[0014] During the model training process, first in the data preparation stage, corresponding labels are constructed according to the number of questions, and the template is instantiated using the clauses in the context text set Context and the category labels of the corresponding questions to construct the K+1-Way Few-Shot dataset training set; then in the model fine-tuning stage, the K+1-Way Few-Shot dataset is used to fine-tune the PLMs.
[0015] Optionally, step 3 includes:
[0016] During the model application process, the fine-tuned PLMs are used to predict the clause categories in the test context text set t_Context, and the similarity between the test question t_query and the categories in the category label Label is calculated. Finally, the clause set corresponding to the category with the maximum similarity to the test question t_query in the category label Label is taken as the answer t_answer.
[0017] Optionally include:
[0018] Given a test context text set t_Context = {clause t1 ,clause t2 ,…,clause tN}, first calculate the similarity between the test question t_query and the various questions in the question set Query, and map it to the category label label t ; Then apply the classification model f NN Predict the category of each clause in the test context text set t_Conext, i.e. pre_label t’ =f NN (clause ti ), clause ti ∈t_Context; finally pre_label t’ ≈labelt The corresponding clause set constitutes the answer set t_answer of t_query = {clause ti}, The value range of tN is a positive integer.
[0019] Optionally, constructing a K+1-Way Few-Shot dataset training set includes: question mapping, mapping the question set Query to the category label Label; data enhancement, using the data enhancement method to expand the answer set Answer to Answer'; template instantiation, using Answer' and the category label Label to instantiate the Template template to obtain the K+1-Way Few-Shot dataset training set.
[0020] Optionally, the problem mapping includes:
[0021] (1) Collect multiple samples of similar problems:
[0022] query i ={query i -sample1,query i -sample2,…,query i -sample n}
[0023] (2) Identify synonyms / near synonyms:
[0024] For each sample query i -sample j , use Word2Vec word embedding to identify synonyms and near-synonyms, for query i -sample j For each word w in , generate a synonym / near-synonym set syn(w);
[0025] (3) Construct the co-occurrence matrix C:
[0026] The initial co-occurrence matrix C = 0, based on each question sample query i -sample j Update the co-occurrence matrix C for each word in w∈query i -sample j and v∈syn(w), C wv =C wv +1;
[0027] Among them, C wv Word w p and w qThe number of co-occurrences in all question samples;
[0028] (4) Extract keywords:
[0029] Count the frequency of each word w in all question samples, that is, the co-occurrence frequency f(w) of word w:
[0030]
[0031] (5) Determine the category label:
[0032] Select the keyword kw with the highest co-occurrence frequency as the question category label label i :
[0033]
[0034] Add category labels for other questions K+1 , map all questions not included in the question set Query to categories.
[0035] Optionally, the data enhancement includes:
[0036] In order to balance the sample sizes of different classes, text data augmentation is used to increase the sample sizes of the first K classes and expand them to the same order of magnitude as the sample sizes of the K+1 classes:
[0037]
[0038] Among them, M'≈|Context-Answer|, a∈[1,K+1];
[0039] First, answer a Put all the clauses of answer into a ', then use synonym / near-synonymous word replacement, masked language model MLM, back translation, sequence generation model Seq2seq, and random exchange method to increase samples, and stop when the total sample size after expansion reaches the requirement;
[0040] In the synonym / near-synonym replacement method, non-stop words in the answer clause are randomly replaced with synonyms / near-synonyms or hypernyms. To prevent large semantic deviations, only one word is replaced each time, and the generated new clause is placed in the answer. a '; In the MLM method, the pre-trained language model RoBerta is used to predict the randomly masked words in the answer clause, and the generated new clause is put into the answer a '; In the back translation method, the answer clause is translated into English by using Chinese-English translation, and then translated back into Chinese to put the generated new clause into the answer a'; Generate semantically consistent new clauses for the answer clause in the Seq2seq model method and put them into the answer a '; If the length of the context text set Context is greater than the answer set Answer; If the above steps still cannot balance the samples, the random exchange method is finally used. Each time the two words in the answer clause are randomly exchanged, the generated new clause is put into the answer a '.
[0041] Optionally, the template instantiation includes:
[0042] Fill the Template template with the clauses in the context text set Context and the sub-labels in the category label Label; first, use answer i 'The answer clause in the ij 'Fill in the placeholder [clause] in the template template, and use the answer accordingly i 'Answered question query i The corresponding label in Label i Fill in the placeholder [label] in the template; then, use the clause clause in the Context-Answer text K+1j Fill in the placeholder [clause] in the template template, and use the label label in the Label accordingly K+1 Fill in the placeholder [label] in the template template; finally, generate the K+1-Way Few-Shot training set trainset = {template(clause aj ',label a ), a∈[1,K+1], j∈[1,iM'].
[0043] In the technical solution provided by the present invention, the method includes defining the question of extractive question answering, defining the context text set Context, the question set Query and the answer set Answer; constructing a model FSPM4WSAEQA based on few-sample prompt learning, and training the model to obtain a classification model f NN ; Apply the obtained classification model f NN , extracting answers from a given context text set based on the question. In a scenario with few samples and in a weak semantic association scenario, this method discovers the association between question and answer pairs for the EQA system through prompt learning, thereby improving the model's answer extraction ability in a weak semantic association scenario. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 A flowchart of a weak semantic association extraction question answering method provided by an embodiment of the present invention;
[0046] Figure 2 A schematic diagram of a model for few-sample prompt learning provided by an embodiment of the present invention;
[0047] Figure 3 A schematic diagram of constructing a training set provided by an embodiment of the present invention;
[0048] Figure 4 A schematic diagram of input coding provided by an embodiment of the present invention;
[0049] Figure 5 A schematic diagram of obtaining answers to questions provided by an embodiment of the present invention;
[0050] Figure 6 A curve chart comparing the semantic associations of different data sets provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.
[0054] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0055] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.
[0056] With the development of Few-Shot Learning (FSL) and Prompt-Based Learning technologies, the present invention analyzes existing data sets from the perspective of the semantic relationship between question-answer pairs and the text context, and observes that although the surface similarity between most question-answer pairs and between the sentence / paragraph contexts of questions and answers is not high, they often have strong semantic associations.
[0057] Figure 1 A flowchart of a weak semantic association extraction question answering method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:
[0058] Step 1: Define the questions of extractive question answering, including the context text set Context, the question set Query, and the answer set Answer.
[0059] In the embodiment of the present invention, step 1 includes:
[0060] The EQA task can be represented as the process of extracting answers from a given context text based on the user's question. Define the context text set Context, the question set Query and the answer set Answer, where Context = {clause1, clause2,…, clause N} represents a context text set consisting of several clauses, and N is a positive integer; Query = {query1, query2, ..., query K} represents a set of K questions raised for the context text set Context, where K is a positive integer; Answer = {answer1, answer2, …, answer K} represents K answer sets extracted from the context text set Context for K questions, and each answer subset answer i ={clause i1 ,clause i2 ,…,clause iM} are all subsets of the context text set Context, i∈[1,K], that is, Therefore, the process of answering questions is the process of finding a suitable subset of answers in the context text set.
[0061] Step 2: Build a model based on few-shot prompting learning (Few-Shot Prompting model for WeakSemantic Associations Extractive Question Answering, FSPM4WSAEQA) and train the model to obtain the classification model f NN .
[0062] In the embodiment of the present invention, Figure 2 As shown, step 2 includes:
[0063] By learning hints, we can enhance the association between weak semantic question-answer pairs and transform them into multi-category classification tasks. i Mapped to category label label i ; and create category labels for questions not included in the question set Query K+1 , all categories form a label set Label = {label1, label2, ..., label K ,label K+1}; Use prompt learning to define the template Template = template ([clause], [label]), and use the few-sample question-answer data to instantiate the template, and then fine-tune the pre-trained language model PLMs to obtain the classification model f NN .
[0064] In the embodiment of the present invention, it specifically includes:
[0065] During the model training process, first in the data preparation stage, corresponding labels are constructed according to the number of questions, and the template is instantiated using the clauses in the context text set Context and the category labels of the corresponding questions to construct the K+1-Way Few-Shot dataset training set; then in the model fine-tuning stage, the K+1-Way Few-Shot dataset is used to fine-tune the PLMs.
[0066] In the embodiment of the present invention, Figure 3 As shown in the figure, constructing the K+1-Way Few-Shot dataset training set includes: question mapping, mapping the question set Query to the category label Label; data enhancement, using the data enhancement method to expand the answer set Answer to Answer'; template instantiation, using Answer' and the category label Label to instantiate the Template template to obtain the K+1-Way Few-Shot dataset training set.
[0067] For a question set Query containing K elements, there are different ways to convert the query into i Mapping to label i When there are not many questions, the K value is small, and the question samples are difficult to collect, you can manually extract the keywords contained in the questions and map them to category labels. When there are many questions and the K value is large, the cost of manual extraction will become extremely high and inefficient, making this method no longer desirable. At this time, you can use the synonym / near-synonym co-occurrence method to extract keywords for multiple question samples of the same type, and use the keyword with the highest co-occurrence frequency as the category label of this question.
[0068] In the embodiment of the present invention, the problem mapping includes:
[0069] (1) Collect multiple samples of similar problems:
[0070] query i ={query i -sample1,query i -sample2,…,query i -sample n}
[0071] (2) Identify synonyms / near synonyms:
[0072] For each sample query i -sample j , use Word2Vec word embedding to identify synonyms and near-synonyms, for query i -sample j For each word w in , generate a synonym / near-synonym set syn(w);
[0073] (3) Construct the co-occurrence matrix C:
[0074] The initial co-occurrence matrix C = 0, based on each question sample query i -sample jUpdate the co-occurrence matrix C for each word in w∈query i -sample j and v∈syn(w), C wv =C wv +1;
[0075] Among them, C wv Word w p and w q The number of co-occurrences in all question samples;
[0076] (4) Extract keywords:
[0077] Count the frequency of each word w in all question samples, that is, the co-occurrence frequency f(w) of word w:
[0078]
[0079] (5) Determine the category label:
[0080] Select the keyword kw with the highest co-occurrence frequency as the question category label label i :
[0081]
[0082] The questions asked about the context text set Context may exceed the K questions in the question set Query, so we add the category labels of other questions K+1 , map all questions not included in the question set Query to categories.
[0083] For an answer set Answer containing K answer subsets, the number of clauses in a single question answer|answer i | is usually much smaller than the number of clauses in the context text |Context|. Even if there are multiple questions in the Query, the number of clauses in the remaining text after removing all the answers is greater than the number of clauses in a single answer, that is, |Context-Answer|>|answer i |. This means that label K+1 The number of samples of the class is more than that of the label i The number of samples in the class (i∈[1,K]) causes imbalance in training samples. Using imbalanced samples may cause the model to converge to the features of the multi-sample class too early, while ignoring the features of the minority class, resulting in poor generalization ability when processing new minority class samples. In addition, the imbalance of sample features when training the model may lead to unbalanced gradient updates, gradient vanishing or gradient exploding, and unstable model training.
[0084] In the embodiment of the present invention, data enhancement includes:
[0085] In order to balance the sample sizes of different classes, text data augmentation is used to increase the sample sizes of the first K classes and expand them to the same order of magnitude as the sample sizes of the K+1 classes:
[0086]
[0087] Among them, M'≈|Context-Answer|, a∈[1,K+1];
[0088] First, answer a Put all the clauses of answer into a ', then use synonym / near-synonymous word replacement, masked language model MLM, back translation, sequence generation model Seq2seq, and random exchange method to increase samples, and stop when the total sample size after expansion reaches the requirement;
[0089] In the synonym / near-synonym replacement method, non-stop words in the answer clause are randomly replaced with synonyms / near-synonyms or hypernyms. To prevent large semantic deviations, only one word is replaced each time, and the generated new clause is placed in the answer. a '; In the MLM method, the pre-trained language model RoBerta is used to predict the randomly masked words in the answer clause, and the generated new clause is put into the answer a '; In the back translation method, the answer clause is translated into English by using Chinese-English translation, and then translated back into Chinese to put the generated new clause into the answer a '; Generate semantically consistent new clauses for the answer clause in the Seq2seq model method and put them into the answer a '; If the length of the context text set Context is greater than the answer set Answer; If the above steps still cannot balance the samples, the random exchange method is finally used. Each time the two words in the answer clause are randomly exchanged, the generated new clause is put into the answer a '.
[0090] In the embodiment of the present invention, instantiating the template includes:
[0091] There are many ways to define templates in prompt learning, such as manual design, automatic search, hard templates, soft templates, hybrid templates, etc. This paper adopts the manual method to design the template Template = template ([clause], [label]), and uses the template to make the relationship between the answer clause and the question label explicit to enhance the semantic association.
[0092] Fill the Template template with the clauses in the context text set Context and the sub-labels in the category label Label; first, use answer i 'The answer clause in the ij 'Fill in the placeholder [clause] in the template template, and use the answer accordingly i 'Answered question query i The corresponding label in Label i Fill in the placeholder [label] in the template; then, use the clause clause in the Context-Answer text K+1j Fill in the placeholder [clause] in the template template, and use the label label in the Label accordingly K+1 Fill in the placeholder [label] in the template template; finally, generate the K+1-Way Few-Shot training set trainset = {template(clause aj ',label a ), a∈[1,K+1], j∈[1,iM'].
[0093] Step 3: Apply the obtained classification model f NN , extract answers from a given context text collection based on the question.
[0094] In the embodiment of the present invention, step 3 includes:
[0095] During the model application process, the fine-tuned PLMs are used to predict the clause categories in the test context text set t_Context, and the similarity between the test question t_query and the categories in the category label Label is calculated. Finally, the clause set corresponding to the category with the maximum similarity to the test question t_query in the category label Label is taken as the answer t_answer.
[0096] In the embodiment of the present invention, it specifically includes:
[0097] Given a test context text set t_Context = {clause t1 ,clause t2 ,…,clause tN}, first calculate the similarity between the test question t_query and the various questions in the question set Query, and map it to the category label label t ; Then apply the classification model f NNPredict the category of each clause in the test context text set t_Conext, i.e. pre_label t’ =f NN (clause ti ), clause ti ∈t_Context; finally pre_label t’ ≈label t The corresponding clause set constitutes the answer set t_answer of t_query = {clause ti}, The value range of tN is a positive integer.
[0098] In the embodiment of the present invention, since its advent, the Transformer network architecture has solved the limitations of traditional network architectures such as Recurrent Neural Network (RNN) and Long Short Term Memory (LSTM) in processing long text sequences by its self-attention mechanism to efficiently capture long-distance dependencies, and continuously breaks through the upper limit of NLP performance in solving various problems, becoming the preferred network architecture in the field of NLP. Transformer includes two frameworks: Encoder and Encoder-Decoder. Among them, the Encoder framework is composed of a multi-layer stacked self-attention mechanism and a feedforward neural network. The self-attention mechanism is used to effectively capture the global dependencies between words in the input text and identify key information related to the problem, so it is more suitable for EQA tasks that deal with factual problems. The Encoder-Decoder framework can better understand the problem and context text through the collaboration of Encoder and Decoder. The Decoder part introduces a conditional generation mechanism, and the context vector passed by the Encoder is combined with the hidden state at the current moment to gradually generate a logically self-consistent new sequence as the answer, so it is more suitable for GQA tasks that require creative answers. The purpose of this invention is to solve the EQA task of weak semantic association question answering. Therefore, the BERT of the Encoder architecture is used as the network model of the system, that is, the classification model f NN =BERT.
[0099] In one possible implementation, Figure 4 As shown, the input code includes:
[0100] Before fine-tuning the model, the training set trainset needs to be formatted and converted into an input representation that satisfies the BERT model. In the above data preprocessing process, the Template template integrates the answer clause and label into one sentence, so the input of the model is a single sentence. It only needs to be in each training sample template (clause aj ',label a ) by adding special marks [CLS] and [SEP] at the beginning and end respectively.
[0101] Like other neural network models, the BERT model cannot process text directly. It needs to use WordPiecetokenizer to decompose the input text into tokens and encode them. The output representation is converted into a high-dimensional vector through Token Embeddings, Segment Embeddings, and Position Embeddings, and then used as the input of the BERT model. In Token Embeddings, the vocabulary of BERT pre-training can be used to map tokens. Since the input is a single sentence, all tokens in the partial embedding are encoded as 0. Position embedding numbers the tokens in sequence. Finally, the attention mask is used to tell the BERT model which tokens should be paid attention to. Since the model needs to predict the label corresponding to the answer, the label i The object that the model wants to focus on is encoded as 1, and the other tokens are encoded as 0.
[0102] In one possible implementation, model fine-tuning includes:
[0103] The model fine-tuning process is to make the classification model f NN The process of predicting the token of the label covered by the attention mask and adjusting the model parameters according to the true label. The process is as follows:
[0104] (1) [MASK] Replacement: Use [MASK] to replace template(clause aj ',label a ) a , get templatem(clause aj ',[MASK]);
[0105] (2) [MASK] prediction: using the classification model f NNPredict the result of [MASK] pre_label a , that is, f NN (templatem(clause aj ',[MASK]))→pre_label a ;
[0106] (3) Parameter tuning: According to label a with pre_label a The deviation between them is adjusted by gradient descent method. NN Parameters, by continuously iteratively reducing the value of the loss function loss to obtain the optimized model.
[0107] In a possible implementation, in the above model fine-tuning optimization, the loss function includes:
[0108] Assume that the training set contains N samples. For each sample (clause aj ',label a ),make Predict pre_label for the model pair [MASK] a The vector representation of y is the sample clause aj 'Corresponding to the real label label a The vector representation of , the loss function loss is the cross entropy loss, and its expression is:
[0109]
[0110] Among them, N represents the number of training samples, K+1 represents the number of labels in Label, and y ba Indicates the true label label a The one-hot representation of Prediction label pre_label a One-hot representation, a∈[1,K+1], b∈[1,N].
[0111] In the embodiment of the present invention, Figure 5 As shown, in the model application phase, given the question t_query, the classification model f is used NN The process of extracting the answer t_answer from the context text t_Context.
[0112] (1) Build a test set:
[0113] Use the given test context text set t_Context = {clauset1, clauset2, ..., clauset tN Each clause inti Fill in the [clause in the template a ]. [label a ] is the result that the model needs to predict, so use [MASK] to replace [label] and build a test set:
[0114] testset = {template1(clause t1 ,[MASK]),template2(clause t2 ,[MASK]),...,
[0115] template tN (clause tN ,[MASK])};
[0116] (2) Label prediction:
[0117] Using the classification model f NN Predict each clause ti The label pre_label ti :
[0118] pre_label ti =f NN (clause ti );
[0119] (3) Test problem mapping:
[0120] Calculate t_query and question set Query = {query1, query2, ..., query K} each type of question sample query i ={query i -sample1,query i -sample2,…,query i -sample n ,}’s semantic similarity.
[0121] First, the BERT model is used to compare t_query and query i -sample j Embedding encoding is performed, which is represented as Bert_Embedding(t_query) and Bert_Embedding(query i -sample j Then, use formula (1) to calculate the cosine similarity between t_query and each sample: cos_sim(t_query,queryi -sample j ):
[0122]
[0123] Use formula (2) to calculate t_query and query i The average similarity of each sample is taken as its similarity with the query. i Semantic similarity:
[0124]
[0125] Find the query with the greatest semantic similarity to t_query t :
[0126]
[0127] Map t_query to query t The corresponding label label t .
[0128] (4) Answer set construction:
[0129] Will be with label t The pre_label with the greatest similarity ti The corresponding clause is the answer:
[0130]
[0131] Finally, the answer set t_answer = {answer t1 ,answer t2 ,...,answer tM}.
[0132] In the embodiment of the present invention, a comparative experiment is performed based on the above method:
[0133] 1. Dataset:
[0134] The performance of the EQA system depends largely on the quality and diversity of the training data. High-quality datasets can not only improve the accuracy of the model, but also help evaluate and compare the effectiveness of different methods. At present, many universities and research institutions have released several EQA datasets for different tasks. For example, the SQuAD (Stanford Question Answering Dataset) dataset released by Stanford University is mainly used for reading comprehension and extractive question answering. It aims to test the model's ability to understand and answer questions from a given paragraph. It includes two versions, 1.1 and 2.0. Among them, the question and answer pairs of the SQuAD1.1 dataset are all derived from Wikipedia articles, covering a wide range of topics and including more than 100,000 question and answer pairs. All questions and answers have been manually annotated and verified to ensure the accuracy and reliability of the data. Later, the SQuAD 2.0 dataset added unanswerable questions to test the model's ability to answer unanswered questions. CMRC (Chinese Machine Reading Comprehension) is a dataset specifically for Chinese reading comprehension tasks. It was jointly released by Harbin Institute of Technology and Alibaba Damo Academy. It aims to simulate reading comprehension tasks in real scenarios and help improve the performance of Chinese machine reading comprehension models. The WebQA dataset is a question-and-answer dataset based on Internet search results. It aims to simulate real-world scenarios in which users seek answers through search engines. Its question-and-answer pairs are derived from real users’ search histories, which makes it more challenging than other datasets. It is available in multiple languages. The English question-and-answer dataset was jointly created by Carnegie Mellon University and Microsoft, while the Chinese question-and-answer dataset was open-sourced by Baidu. The data comes from Baidu Knows and is divided into manual annotations and browser retrieval. The TriviaQA dataset is a large open-domain question-and-answer dataset released by the University of Washington. It is designed to evaluate the performance of machines in answering common-sense and factual questions. It contains tens of thousands of questions, and the answers come from Wikipedia, books, websites, etc., which increases the diversity and complexity of the dataset. Figure 6 As shown in the figure, the analysis found that the current commonly used datasets have strong semantic associations between question and answer pairs and between the sentence / paragraph contexts of questions and answers, and are not suitable for the analysis of weak semantic association problems.
[0135] In the experiment of the present invention, three types of documents, "Project Application Notice", "Enterprise Reward Policy" and "Bidding Documents", were collected from the industry competitive intelligence analysis library (twofat.net) developed by the present invention as data sets. Each data set contains 200 documents. In order to reduce labor costs, only 20 documents are allocated as the context text Context of the small sample training set trainset, and 180 documents are allocated as the context text t_Context of the test set testset. Through user log analysis, the questions asked by users on the three types of context texts are obtained and thematic clustering is performed to obtain the questions with the highest user attention, and then the questions with high semantic relevance to the answers are removed. Finally, several high-frequency common weak semantically associated question-answer pairs are sorted out for the three types of documents as data sets.
[0136] In order to examine the semantic relevance between question-answer pairs in different datasets, we compared the three datasets we constructed with five commonly used datasets. Application, Reward, and Bid are used to represent the three self-built datasets of "Project Application Notice", "Enterprise Reward Policy", and "Bidding Documents" respectively. The five commonly used datasets are SQuAD 2.0, CMRC2018, WebQA-E, WebQA-C, and TriviaQA datasets, where WebQA-E and WebQA-C are the English and Chinese versions of WebQA datasets respectively.
[0137] In the analysis, the BERT model is used to embed the question-answer pairs, which are represented as Bert_Embedding(question i ) and Bert_Embedding(answer i ), and then use formula (1) to calculate the cosine similarity cos_sim(question i ,answer i ), obtain the semantic similarity between question and answer pairs. Analysis results, such as Figure 6 As shown in the figure, the horizontal axis represents the semantic similarity between the question and answer pairs in the dataset and between the sentence / paragraph contexts of the questions and answers, and the vertical axis represents the proportion of question and answer pairs with different similarities in the dataset.
[0138] Depend on Figure 6It can be seen that the semantic relevance between question-answer pairs in the five commonly used datasets is significantly stronger than that in the three self-built datasets. In the commonly used datasets, the similarity between most question-answer pairs is higher than 0.55 and lower than 0.85 (except for WebQA-E). This is because the question is an interrogative sentence, while the answer is a descriptive or declarative sentence. The two share certain background context information, so they have a certain semantic relevance, but usually do not have complete semantic similarity. In the three self-built datasets, the similarity between most question-answer pairs is lower than 0.55, and the lower similarity also proves that the semantic relevance between the two is weaker.
[0139] 2. Experimental evaluation indicators:
[0140] To evaluate the experimental results of the proposed method, three indicators, namely precision P, recall R and F1 value, are used to evaluate the effect. The three indicators are calculated using formulas (3), (4) and (5) respectively.
[0141]
[0142] Among them, TP represents the query i The number of answers retrieved from the answers, FP means that the retrieval result is misidentified as the question query i The number of answers, FN represents the question query i The number of answers that were not retrieved.
[0143] 3. Baseline comparison model:
[0144] This paper takes two commonly used solutions of current question-answering systems: Bert Model for Extractive Question Answering (BertM4EQA) based on the BERT model and GPT Model for Extractive Question Answering (GPTM4EQA) as benchmark methods for comparison with the proposed FSPM4WSAEQA model.
[0145] (1) BertM4EQA
[0146] The typical method of using the BERT pre-trained model to solve the downstream task EQA is to add a linear layer as a classifier on the basis of the BERT network model. In this solution, this paper fine-tunes the BERT model with a linear classifier added by the trainset dataset, and then uses the classifier to predict the start and end positions of the answer corresponding to the question in the testset dataset in the context text t_Context to obtain the answer.
[0147] (2)GPTM4EQA
[0148] The GPT generative model is combined with prompt learning. Prompts can also enable the model to answer questions within a limited range to solve EQA tasks. The key lies in the design of prompts. In this solution, prompts are carefully designed manually, and the context text and questions in the trainset dataset are submitted to the GPT model together with the prompts. The questions are clearly described to guide the model to focus on relevant information in the context. The answers extracted by the model are then compared with the correct answers to continuously improve the prompts, and finally suitable prompts are found to test the question and answer on the testset dataset.
[0149] (3) FSPM4WSAEQA
[0150] In the FSPM4WSAEQA model proposed in the present invention, the model adopts the Bert model open sourced by Google for Chinese training, that is, f NN = BertGoogle; template template is defined as: data set name + question mapping label + answer clause pattern, such as the template template1 of the project application notice data set is: [label i ]Includes: [clause ij ]; use the trainset dataset to fine-tune the FSPM4WSAEQA model; finally use the fine-tuned FSPM4WSAEQA model to predict the answers to the questions in the testset dataset.
[0151] 4. Comparison results and analysis:
[0152] The average precision P, recall R and F1 value of the test results of the above three models on three types of self-built data sets are shown in Table 1.
[0153] Table 1
[0154]
[0155] It can be seen from the experimental results that the FSPM4WSAEQA model can significantly improve the quality of weak semantic association question and answer results. Overall, its F1 value is about 20 percentage points higher than that of the BertM4EQA model and more than 10 percentage points higher than that of the GPTM4EQA model.
[0156] By looking at the answers given by the BertM4EQA model, we can find that the model can effectively find answers in the context text that are strongly related to the question semantics, and most of the answers it gives are correct answers, that is, the FP value in formula (3) is usually small, which makes it have a higher precision P. Its P value results are mostly similar to the precision P value of the FSPM4WSAEQA model, or even slightly higher. However, since it cannot return answers that are weakly related to the question semantics, it will miss some correct answers, that is, the FN value in formula (4) is usually large, resulting in a lower R value, resulting in a lower F1 value. Therefore, this model has a large missed detection rate.
[0157] By looking at the answers given by the GPTM4EQA model, we can find that, first of all, the GPT generative model can give answers based on the questions asked, but even if the designed prompt requires the model to use the original text content when answering, when the context text contains the answer, the answer it gives is very similar to the correct answer, but there are still many "free plays", which affect the credibility of the answer, especially when the context text is a legal or policy text, the approximate answer it gives may lead to wrong results. At the same time, these "plays" will also reduce the TP value in formula (3), making the precision P value lower than the precision P value of the FSPM4WSAEQA model. When there is no answer in the context text, the GPTM4EQA model will also give a "so-called" answer, which affects the TP value in formula (4), causing the recall R value to be lower than the recall R value of the FSPM4WSAEQA model. Secondly, if the length of the document exceeds the allowed length of a single input, slicing it for multiple answers will also affect the TP value in formulas (3) and (4), causing the P value and R value to be lower than the results of the FSPM4WSAEQA model. In particular, for very long bidding documents, more invalid results will be obtained after multiple answers are output, which seriously affects the credibility of the answer. Finally, even if the GPT model has a self-attention mechanism, when the length of the context text far exceeds the prompt, the model's attention will be distracted, the attention to the prompt will be weakened, and the "willingness" to follow the prompt instructions will be greatly reduced, generating more random content in the answer.
[0158] By looking at the answers given by the FSPM4WSAEQA model, we can find that, first, by using templates to make the association between question-answer pairs explicit and using a few-sample fine-tuning training, the model can learn the association between answers and specific semantic labels, and then understand the weak semantic relationship between answers and questions, so that it can find answer clauses with weak semantic associations in the context text according to the questions. Furthermore, the model uses a classification method instead of using the cloze ability of the BERT model itself, which can control the mapping of answers to labels within a small range of options rather than choosing from the entire vocabulary, and also make weak semantic relationships more meaningful when classified within a limited range, so that their impact on the TP value in public (3) and (4) far exceeds the FP value and FN value, thereby obtaining better precision P and recall R.
[0159] In the technical solution provided by the present invention, the method includes defining the question of extractive question answering, defining the context text set Context, the question set Query and the answer set Answer; constructing a model FSPM4WSAEQA based on few-sample prompt learning, and training the model to obtain a classification model f NN ; Apply the obtained classification model f NN , extracting answers from a given context text set based on the question. In a scenario with few samples and in a weak semantic association scenario, this method discovers the association between question and answer pairs for the EQA system through prompt learning, thereby improving the model's answer extraction ability in a weak semantic association scenario.
[0160] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A weak semantic association extractive question answering method based on few-sample prompt learning, characterized in that: The method comprises: Step 1: Define the questions of extractive question answering, including the context text set Context, the question set Query and the answer set Answer; Step 2: Build the model FSPM4WSAEQA based on few-sample prompt learning and train the model to obtain the classification model f NN ; Step 3: Apply the obtained classification model f NN , extract answers from a given context text collection based on the question.
2. The method according to claim 1, characterized in that The step 1 comprises: Define the context text set Context, the question set Query and the answer set Answer, where Context = {clause1, clause2,…, clause N } represents a context text set consisting of several clauses, and N is a positive integer; Query = {query1, query2, ..., query K } represents a set of K questions raised for the context text set Context, where K is a positive integer; Answer = {answer1, answer2, …, answer K } represents K answer sets extracted from the context text set Context for K questions, and each answer subset answer i ={clause i1 ,clause i2 ,…,clause iM } are all subsets of the context text set Context, i∈[1,K], that is, 3. The method according to claim 1, characterized in that The step 2 comprises: Each type of question query in the question set Query i Mapped to category label label i ; and create category labels for questions not included in the question set Query K+1 , all categories form a label set Label = {label1, label2, ..., label K ,label K+1 }; Use prompt learning to define the template Template = template ([clause], [label]), and use the few-sample question-answer data to instantiate the template, and then fine-tune the pre-trained language model PLMs to obtain the classification model f NN .
4. The method according to claim 3, characterized in that include: During the model training process, first in the data preparation stage, corresponding labels are constructed according to the number of questions, and the template is instantiated using the clauses in the context text set Context and the category labels of the corresponding questions to construct the K+1-Way Few-Shotdataset training set; then in the model fine-tuning stage, the K+1-Way Few-Shot dataset is used to fine-tune PLMs.
5. The method according to claim 1, characterized in that The step 3 comprises: During the model application process, the fine-tuned PLMs are used to predict the clause categories in the test context text set t_Context, and the similarity between the test question t_query and the categories in the category label Label is calculated. Finally, the clause set corresponding to the category with the maximum similarity to the test question t_query in the category label Label is taken as the answer t_answer.
6. The method according to claim 5, characterized in that include: Given a test context text set t_Context = {clause t1 ,clause t2 ,…,clauset tN }, first calculate the similarity between the test question t_query and the various questions in the question set Query, and map it to the category label label t ; Then apply the classification model f NN Predict the category of each clause in the test context text set t_Conext, i.e. pre_label t’ =f NN (clause ti ), clause ti ∈t_Context; finally pre_label t’ ≈label t The corresponding clause set constitutes the answer set t_answer of t_query = {clause ti }, The value range of tN is a positive integer.
7. The method according to claim 4, characterized in that Constructing the K+1-Way Few-Shot dataset training set includes: question mapping, mapping the question set Query to the category label Label; data enhancement, using the data enhancement method to expand the answer set Answer to Answer'; template instantiation, using Answer' and the category label Label to instantiate the Template template to obtain the K+1-Way Few-Shot dataset training set.
8. The method according to claim 7, characterized in that The problem mapping includes: (1) Collect multiple samples of similar problems: query i ={query i -sample1,query i -sample2,…,query i -sample n } (2) Identify synonyms / near synonyms: For each sample query i -sample j , use Word2Vec word embedding to identify synonyms and near-synonyms, for query i -sample j For each word w in , generate a synonym / near-synonym set syn(w); (3) Construct the co-occurrence matrix C: The initial co-occurrence matrix C = 0, based on each question sample query i -sample j Update the co-occurrence matrix C for each word in w∈query i -sample j and v∈syn(w), C wv =C wv +1; Among them, C wv Word w p and w q The number of co-occurrences in all question samples; (4) Extract keywords: Count the frequency of each word w in all question samples, that is, the co-occurrence frequency f(w) of word w: (5) Determine the category label: Select the keyword kw with the highest co-occurrence frequency as the question category label label i : Add category labels for other questions K+1 , map all questions not included in the question set Query to categories.
9. The method according to claim 7, characterized in that: The data enhancement includes: In order to balance the sample sizes of different classes, text data augmentation is used to increase the sample sizes of the first K classes and expand them to the same order of magnitude as the sample sizes of the K+1 classes: Among them, M'≈|Context-Answer|, a∈[1,K+1]; First, answer a Put all the clauses of answer into a ', then use synonym / near-synonymous word replacement, masked language model MLM, back translation, sequence generation model Seq2seq, and random exchange method to increase samples, and stop when the total sample size after expansion reaches the requirement; In the synonym / near-synonym replacement method, non-stop words in the answer clause are randomly replaced with synonyms / near-synonyms or hypernyms. To prevent large semantic deviations, only one word is replaced each time, and the generated new clause is placed in the answer. a '; In the MLM method, the pre-trained language model RoBerta is used to predict the randomly masked words in the answer clause, and the generated new clause is put into the answer a '; In the back translation method, the answer clause is translated into English by using Chinese-English translation, and then translated back into Chinese to put the generated new clause into the answer a '; Generate semantically consistent new clauses for the answer clause in the Seq2seq model method and put them into the answer a '; If the length of the context text set Context is greater than the answer set Answer; If the above steps still cannot balance the samples, the random exchange method is finally used. Each time the two words in the answer clause are randomly exchanged, the generated new clause is put into the answer a '.
10. The method according to claim 7, characterized in that The template instantiation includes: Fill the Template template with the clauses in the context text set Context and the sub-labels in the category label Label; first, use answer i 'The answer clause in the ij 'Fill in the placeholder [clause] in the template template, and use the answer accordingly i 'Answered question query i The corresponding label in Label i Fill in the placeholder [label] in the template; then, use the clause clause in the Context-Answer text K+1j Fill in the placeholder [clause] in the template template, and use the label label in the Label accordingly K+1 Fill in the placeholder [label] in the template template; finally, generate the K+1-Way Few-Shot training set trainset = {template(clause aj ',label a ), a∈[1,K+1],j∈[1,iM']}.
Citation Information
Cited By
Tourism knowledge graph multi-dimensional construction method and system of deep learning hybrid model
CN120541234A
