A two-stage attention-based multi-feature fusion problem generation method and system

Abstract

A two-stage attention-based multi-feature fusion question generation method and system, the multi-feature fusion question generation method comprising inputting context into a multi-feature encoder, embedding a feature vector by the multi-feature encoder, and adopting two-stage attention encoding; inputting context into a question type encoder, predicting and encoding a question type; inputting two encodings obtained by the multi-feature encoder and the question type encoder into a question decoder by adopting a cross-attention mechanism, and outputting a generated question by the question decoder. The application uses a Bi-LSTM model to extract context features, combines a soft attention matrix, places the attention center of the model on the part related to the answer, adopts a gated self-attention mechanism to filter and integrate features, solves the long-distance dependence problem of text, introduces a question word prediction guide mechanism to guide the generation of questions, and can generate more accurate and relevant context information vectors.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer text processing, and particularly relates to a multi-feature fusion question generation method and system based on two-stage attention. BACKGROUND

[0002] Question generation is an important research direction in the field of natural language processing, and its main purpose is to automatically ask questions about a text and generate questions related to the context and having answerability. Question generation has wide application prospects in the field of education, as it can help teachers create more teaching questions to improve students' learning effect. At the same time, it can also help students better understand the course content and promote their in-depth thinking and research in the knowledge field.

[0003] In addition, question generation also has a positive role in promoting the depth of data usage, such as data augmentation to alleviate the real problem of insufficient question and answer corpus, application of dialogue task, etc. Although the existing question generation model can generate questions of a certain quality, the processing of complex language expression and long text paragraphs has not been completely solved. Therefore, it is necessary to further improve the accuracy and fluency of question generation technology to better promote the development of the field of intelligent education, improve the natural language reasoning ability, and provide support for question and answer tasks and other technical research. SUMMARY

[0004] The purpose of the application is to provide a multi-feature fusion question generation method and system based on two-stage attention, which uses soft attention and self-attention mechanisms to process paragraph-answer and paragraph data respectively, and integrates multiple features to enhance semantic representation and improve the accuracy and fluency of generated questions.

[0005] In order to achieve the above purpose, the application has the following technical solutions:

[0006] A multi-feature fusion question generation method based on two-stage attention, comprising:

[0007] inputting the context into a multi-feature encoder, embedding a feature vector by the multi-feature encoder, and using two-stage attention encoding;

[0008] inputting the context into a question type encoder, predicting and encoding the question type;

[0009] using a cross-attention mechanism to fuse the two encodings obtained by the multi-feature encoder and the question type encoder into a question decoder, and outputting the generated question by the question decoder.

[0010] As a preferred scheme, the multi-feature encoder, the question type encoder and the decoder are established based on the encoding-decoding architecture of the Seq2Seq model.

[0011] As a preferred scheme, in the step of embedding the feature vector by the multi-feature encoder, the words in the context sentence are converted into corresponding pre-trained word vectors, and the pre-trained word vectors are input into the multi-feature encoder while considering language features and sentiment features, semantic role labeling features and dependency syntax features, and the language features include named entity, part of speech and case features.

[0012] As a preferred scheme, the input of the multi-feature encoder includes word embedding, answer label feature, part of speech feature, named entity feature and sentence feature where the answer is located; wherein the word embedding: adopts pre-trained GloVe embedding representation, learns word vectors using global co-occurrence statistical information, and each word in the text is represented by a corresponding word embedding vector feature through GloVe;

[0013] The answer label feature: each word in the paragraph text is labeled, and the BIO label method is used, where label B represents the beginning of the answer, label I represents a continuous part of the answer, and label O marks a word that does not constitute a part of the answer;

[0014] The part of speech feature: refers to the attribute used to describe the grammatical function of a word in a natural language processing task, and the One-Hot one-hot encoding is used for the part of speech feature, and then the feature is extracted by the encoder;

[0015] The named entity feature: refers to the attribute used to describe a word or phrase with a specific entity name in a natural language processing task, and the One-Hot one-hot encoding is also used for the named entity;

[0016] The sentence feature where the answer is located: describes the sentence containing the correct answer in the text, and if a sentence in the paragraph contains an answer fragment, all the words in the corresponding sentence are marked as 1, and the words in other sentences that do not contain the answer are marked as 0.

[0017] As a preferred scheme, the multi-feature encoder uses a Bi-LSTM model for encoding, which is composed of two LSTM layers based on the RNN structure, one of which processes the input sequence from the beginning to the end, and the other processes the input sequence from the end to the beginning; the outputs of the two directions are connected at each time to form the final output of the Bi-LSTM model;

[0018] The input received by the Bi-LSTM model includes the answer a, the paragraph P where the answer belongs, and the multiple feature combinations mentioned above; the embedding vectors of the answer a and the paragraph P where the answer belongs are and respectively.

[0019]

[0020]

[0021] In the formula, w i For word vector features; n i Name entity vector features; p i As part-of-speech feature vector; i The sentence containing the answer has the following characteristics; a i Answer label vector features; symbols; concatenation operations indicating concatenation methods; The embedded representation of the answer; Let i be the embedded representation of the paragraph P to which the answer belongs; i represents the i-th word in the text.

[0022] The embedding vector obtained from answer a and the paragraph P to which the answer belongs. The vectors are passed to a shared Bi-LSTM layer and encoded to obtain the state vector representation at each time t.

[0023]

[0024]

[0025] In the formula, h t-1 This represents the state at the previous moment;

[0026] Each time step t of the Bi-LSTM layer generates two hidden vector sequences, namely the forward sequence and the backward sequence. The hidden vector sequence at each time step t is then represented as h. i :

[0027]

[0028] In the formula, These represent the hidden vector representations of the forward and backward sequences at each time t, respectively; h t The final hidden vector representation at each time step t is obtained by concatenating the preceding and following sequences.

[0029]

[0030]

[0031] In the formula, H A H P This is the final state representation matrix.

[0032] As a preferred solution, the first stage attention text in the two-stage attention encoding adopts a soft attention matching matrix, and the attention matrix of the paragraph context to the answer is obtained by calculating the mutual connection between the answer representation and the context representation:

[0033]

[0034] In the formula, is the i-th word vector representation in the paragraph; is the j-th word vector representation in the answer; M ij is a result element in the attention matrix;

[0035] F(x)=σ(Wx+b)

[0036] In the formula, F(·) is a standard nonlinear transformation function, and sigma is a Sigmoid function; W is a parameter matrix;

[0037] A maximum pooling operation in the column direction of the matrix M is adopted to generate an attention vector:

[0038] a 1 =softmax(max(M))

[0039] In the formula, a 1 is the attention weight score obtained; M is a soft attention matching matrix; each element is based on the maximum correlation between the i-th word in the paragraph P and the answer A to capture the relevance of the i-th word in the paragraph P to the answer A;

[0040] The attention score is applied to H P to obtain a paragraph representation C with a relevant weight coefficient:

[0041] C=H p ⊙a 1

[0042] In the formula, represents element-wise multiplication, and C is a vector representation after the first stage attention, in which the words directly related to the answer are amplified when the attention score is high, and the irrelevant words are filtered out when the attention score is low.

[0043] As a preferred solution, the second stage attention mechanism in the two-stage attention encoding adopts a gated self-attention fusion method;

[0044] The vector representation C obtained by the first stage attention is taken as input, and the input itself is calculated to obtain a self-matching representation by attention matching. The attention score of the state vector at each time t is calculated as follows:

[0045]

[0046] where c t is the state representation of the one-stage attention at each time t; W S is a parameter matrix; the computed attention score a 2 is multiplied with the vector representation C obtained from the one-stage attention to obtain the two-stage attention state representation:

[0047]

[0048] where s t is the current state representation computed through attention weights at each time t; W

[0049] is the final self-matching representation matrix;

[0050] The current state representation s t computed through attention weights at each time t is combined with the state representation c t after one-stage attention at each time t to obtain a new self-matching enhanced representation f t , g t is a learnable gate vector representation that determines the final state representation from f t , g t The computation expression is as follows:

[0051] f t = tanh(W f [c t ,s t ])

[0052] g t = sigmoid(W g [c t ,s t ])

[0053]

[0054] where W f , W g are parameter matrices, respectively;

[0055]

[0056] ​As a preferred solution, the question type encoder counts the questions marked by human in the SQuAD1.1 question answering dataset, and the question words include How, Why, Who, Which, Where, When, What, Was, Has and Did, and other question word types are classified as Other, and the question word types with less quantity are supplemented from the NewsQA dataset;

[0057] The context information, the answer information, the answer named entity information and the answer part-of-speech information are taken as a set of training data, the number of each category is balanced, the prediction task is classified as a typical text classification task, and a fine-tuned BERT model is used as a classification model;

[0058] Each element in the context information, the answer information, the answer named entity information and the answer part-of-speech information of each input sample of the BERT model is converted into a corresponding token sequence, and the SPE symbol is used to separate each element; a CLS symbol is added in front of the input sequence to generate a vector representation of the sequence:

[0059] S=[ <cls>,context, <sep>, answer, <sep>,ner, <sep>pos]

[0060] wherein, context is the sentence information where the answer is located; answer is the answer information; ner is the named entity information of the answer; and pos is the part-of-speech information of the answer. <sep>is a separator;

[0061] Then the BERT model is used to extract features from the sequence:

[0062] X hidden = BERT(S)

[0063] The first position of the hidden layer CLS is used to extract features using a fully connected layer. The feature vector includes all semantic information. The fully connected layer reduces the dimension of the CLS feature vector to the number of labels t, and the size of t is the number of question type categories:

[0064] Y = Linear t (X hidde )

[0065] The Softmax function is used to calculate the corresponding label, and the final output result is the label y:

[0066] y = softmax(Y)

[0067] The question type encoder uses a Bi-LSTM model for encoding, taking the answer a and the question type qt as input, where qt is obtained from the label y predicted by the question type, and the embedding vector representation of the answer and question type is x q :

[0068] x q = [a, qt]

[0069] In the formula, x q is an embedding vector; a is an answer vector feature; qt is a question type vector feature; and the symbol represents a vector concatenation operation in parallel mode;

[0070] x q is passed to the Bi-LSTM layer to obtain the state vector representation at each time t:

[0071]

[0072] The Bi-LSTM layer produces two hidden vector sequences at each time t, namely the forward sequence and the backward sequence. The hidden vector sequence at each time t is represented as:

[0073]

[0074] In the formula respectively represent the hidden vector representation of the forward and backward sequences at each time t; and h t is the final hidden vector representation at each time t obtained by concatenating the forward and backward sequences:

[0075]

[0076] where H T represents the final obtained hidden vector representation matrix, and L is the input length of the answer information and the question type.

[0077] As a preferred solution, the question decoder decodes the output sequence of the multi-feature encoder by using a unidirectional LSTM model, generates a corresponding target sequence step by step by using the information sequence generated by the multi-feature encoder as an input sequence and using a unidirectional LSTM decoder, uses the last hidden layer state of the question type decoder as the initialization state u0 of the decoder, and generates the hidden layer state of the decoder at each time t by using the hidden layer state u T-1 and the last predicted information y t-1 in the output sequence.

[0078] u t = LSTM(u t-1 , y t-1 )

[0079] The attention score r T is calculated by re-considering all the hidden layer state vectors of the input sequence each time the hidden layer state is updated.

[0080] r T = H s · W T u d t

[0081] where H S is the final state representation of the multi-feature encoder, W d is a parameter matrix, and u t is the hidden state representation of the decoder at time t.

[0082] The attention score is normalized by using a softmax function to obtain the attention weight at time t:

[0083] a t = softmax(r t )

[0084] The final decoding state at time t is obtained by using the original decoder hidden layer state vector u t at time t and the attention-weighted state vector c t .

[0085] c t = H d · a t

[0086]

[0087] In the decoding stage, a copying mechanism based on the maximum output pointer and a pointer generation network are adopted, and through the calculation of the probability of copying words from the input and the probability of generating words from the vocabulary, the final sentence result is obtained;

[0088] The probability of generating from the vocabulary in the generation process is:

[0089]

[0090] A strategy based on the maximum output pointer is adopted, when a repeated word appears, only the attention weight corresponding to the maximum word is selected as the final weight distribution, and the word not appearing in the input sequence is set to negative infinity:

[0091]

[0092] The probability P copy (w) of copying the original text in the generation process is:

[0093] P copy (w)=softmax(W v sc t )

[0094] The final generation probability P gen (w) is represented as:

[0095] P gen (w)=P copy (w)+P vocab (w)。

[0096] A two-stage attention-based multi-feature fusion question generation system, comprising:

[0097] A feature vector embedding module is used to input a context into a multi-feature encoder, embed a feature vector by the multi-feature encoder, and adopt two-stage attention encoding;

[0098] A question type prediction module is used to input a context into a question type encoder, predict a question type and encode;

[0099] A question decoding module is used to input two encodings obtained by the multi-feature encoder and the question type encoder into a question decoder by adopting a cross-attention mechanism, and output a generated question by the question decoder.

[0100] Compared with the prior art, the present application has at least the following beneficial effects:

[0101] When the text is input, firstly, the multi-feature encoder embeds the feature vector, then adopts two-stage attention encoding, and the question type encoder predicts the question type and encodes, then the two encodings adopt the cross attention mechanism to be fused into the question decoder, and finally the generated question is input. The question generation method of the application obtains better generation performance by using the multi-feature encoder, the question type encoder and the question decoder, and introducing the two-stage attention mechanism. Through the question generation method proposed in the application, the text noise and long distance dependence problem can be better solved, and the question generation quality is improved.

[0102] Further, the question generation method of the application uses soft attention and self-attention mechanisms to process paragraph-answer and paragraph respectively, and integrates various features to enhance semantic representation. Specifically, the method of the application uses a Bi-LSTM model to extract context features, and combines a soft attention matrix to place the attention center of the model on the part related to the answer to filter irrelevant noise information. Then, a gated self-attention mechanism is used to further filter and integrate these features, solve the long distance dependence problem of text, and generate more accurate and relevant context information vectors. At the same time, the method of the application also introduces a question word prediction guidance mechanism to guide the generation of questions, further improving the accuracy and fluency of the questions. BRIEF DESCRIPTION OF DRAWINGS

[0103] In order to more clearly illustrate the technical solutions in the present application, the drawings needed to be used in the application description will be briefly introduced below, and the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the premise of not paying creative labor.

[0104] Figure 1 The embodiment of the application is based on a two-stage attention multi-feature fusion question generation method model framework diagram.

[0105] Figure 2 The embodiment of the application is based on a two-stage attention multi-feature fusion question generation method flowchart.

[0106] Figure 3 The model framework diagram of the multi-feature word vector embedding of the embodiment of the application. DETAILED DESCRIPTION

[0107] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be described and explained below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0108] Referring to Figure 1 The application is based on a two-stage attention-based multi-feature fusion question generation method based on the encoding-decoding architecture of the Seq2Seq model, including a multi-feature encoder, a question type encoder and a decoder.

[0109] Referring to Figure 2 The two-stage attention-based multi-feature fusion question generation method of the embodiment of the application comprises:

[0110] The context is input into the multi-feature encoder, the feature vector is embedded by the multi-feature encoder, and two-stage attention encoding is adopted;

[0111] The context is input into the question type encoder, the question type is predicted and encoded;

[0112] The two encodings obtained by the multi-feature encoder and the question type encoder are fused by using a cross-attention mechanism and input into a question decoder, and the question generated by the question decoder is output.

[0113] In a possible implementation, the feature vector embedding process of the multi-feature encoder is as follows:

[0114] In a traditional model encoder, it is usually necessary to convert the words in the input sentence into corresponding pre-trained word vectors, and input these word vectors into the encoder to model and represent the sentence. In addition to the basic pre-trained word vectors, some specific language features need to be considered in natural language processing tasks to improve the accuracy of task completion. These language features include named entities, word forms and case features.

[0115] Named entities refer to words with specific meanings in text, such as names, place names, organization names, etc. Word forms represent the grammatical function of words in a sentence, such as nouns, verbs, adjectives, etc. And the case feature indicates whether the first letter of the word is capitalized, which is very helpful for distinguishing proper nouns. In addition, there are some other features that can be used in the model encoder, such as sentiment features, semantic role labeling features and dependency syntax features, etc. These features can help the model better understand the meaning and grammatical structure of the text, thereby improving the performance and accuracy of the model.

[0116] As Figure 3 shown, the input of the multi-feature encoder includes word embedding, answer label feature, word form feature, named entity feature and answer sentence feature.

[0117] Word Embedding: The pre-trained GloVe embedding representation is adopted. One of the advantages of GloVe over other word vector models is that it utilizes global co-occurrence statistics to learn word vectors, rather than just local context windows. This allows it to capture more extensive semantic and syntactic relationships in word vectors. Each word in the text is given its own word embedding vector feature through GloVe.

[0118] Answer Marking Feature: This feature marks each word in the passage text. The marking method adopted by the present application is the BIO tag method, i.e. the label B represents the beginning of the answer, the label I represents a continuous part of the answer, and the label O marks the words that do not constitute part of the answer. Through BIO tagging, the answer position is encoded as a hidden vector and used to guide the model to generate more relevant questions to the answer.

[0119] Part-of-Speech Feature: Part-of-speech feature refers to the attribute used to describe the grammatical function of a word in natural language processing tasks, such as noun, verb, adjective, adverb, etc. It is usually used to analyze and label words in a sentence to help identify the grammatical structure and meaning of the sentence. The present application uses One-Hot encoding for part-of-speech features, and then extracts features through an encoder.

[0120] Named Entity Feature: Named entity feature refers to the attribute used to describe words or phrases with specific entity names in text in natural language processing tasks. These entities can be names of people, places, organizations, time, date, etc. with special meaning. The same One-Hot encoding method is used for named entities.

[0121] Answer Sentence Feature: Answer sentence feature refers to the feature of the sentence containing the correct answer in the text. If a sentence in the passage contains an answer fragment, then mark all the words in that sentence as 1, and mark the words in other sentences that do not contain the answer as 0. The purpose of this feature is similar to the answer marking feature, which makes the model pay more attention to the information in the sentence containing the answer.

[0122] The multi-feature encoder adopts Bi-LSTM (Bidirectional Long Short-Term Memory) as the encoder for encoding. It is composed of two LSTM layers based on the RNN structure, one processes the input sequence from start to end (forward direction), and the other processes the input sequence from end to start (reverse direction). The outputs of the two directions are concatenated at each time step to form the final output of the Bi-LSTM. It can capture the forward and reverse context information of the sequence, thus better modeling the long-distance dependencies in the input sequence. The input accepted by the model includes: answer a, paragraph P to which the answer belongs, and the multiple feature combinations mentioned above. From the previous subsection, the final embedding vectors of the answer and the paragraph are

[0123]

[0124]

[0125] In formula (1-1) and (1-2), w i is the word vector feature; n i is the named entity vector feature; p i is the part-of-speech feature vector; as i is the answer sentence feature; a i is the answer label vector feature; the symbol ";” is the concatenation operation in series; is the final embedding representation of the answer; is the final embedding representation of the paragraph; i represents the i-th word in the text.

[0126] After that, they are respectively passed to the shared Bi-LSTM layer, and the state vector representation at each time step t is obtained after encoding

[0127]

[0128]

[0129] In formula (1-3) and (1-4), h t-1 is the state representation of the previous moment.

[0130] Finally, each time step t of the Bi-LSTM produces two hidden vector sequences, i.e. the forward sequence and the backward sequence, as shown in formula (1-5), the hidden vector sequence at each t time is represented as h i :

[0131]

[0132] In formula (1-5), respectively represent the hidden vector representation of the forward and backward sequence at time t; h t is the final hidden vector representation at time t obtained by concatenating the forward and backward sequence.

[0133]

[0134]

[0135] In formula (1-6) and (1-7), H A , H P is the final obtained state representation matrix.

[0136] a. First-order attention

[0137] In order to make the model pay more attention to the context directly related to the answer and filter some noise information irrelevant to the answer, the first-order attention adopted by the present application is a soft attention matching matrix, that is, by calculating the mutual relationship between the answer representation and the context representation, the attention matrix of the paragraph context to the answer is obtained.

[0138]

[0139] In formula (1-8), is the i-th word vector representation in the paragraph; is the j-th word vector representation in the answer; M is a soft attention matching matrix; M ij is a result element in the attention matrix.

[0140] F(x) = σ(Wx + b) (1-9)

[0141] In formula (1-9), F(·) is a standard nonlinear transformation function (i.e. F(x) = σ(Wx + b), where σ is a Sigmoid function); W is a parameter matrix.

[0142] Next, the maximum pooling operation in the column direction of the matrix M is adopted, so that an attention vector can be generated:

[0143] a 1 = softmax(max(M)) (1-10)

[0144] In formula (1-10), a 1 is the obtained attention weight score; M is a soft attention matching matrix; each element is based on the maximum correlation between the i-th word in the paragraph P and the answer A to capture the relevance of the i-th word in the paragraph P to the answer A.

[0145] After that, the present application applies the attention score to H P , to obtain the paragraph representation C with relevant weight coefficients:

[0146] C = H p ⊙a 1 (1-11)

[0147] In formula (1-11), ⊙ represents element-by-element multiplication, and C is a semantic representation after the first-stage attention, in which words directly related to the answer are amplified when the attention score is high, and irrelevant words are filtered out when the attention score is low.

[0148] b. Second-order attention

[0149] The second-stage attention mechanism adopts a gated self-attention fusion method, aiming to further improve the model's ability to model the relationship between the context and the answer. This method is based on the traditional self-attention mechanism and introduces a gating mechanism to optimize the distribution of attention weights by controlling the weight of each attention and the interaction between different attentions, thereby achieving more refined feature fusion. In addition, this method can also effectively solve the long-distance dependency problem. By modeling the interaction between attentions, it can better capture the complex interaction between the context and the answer, thereby improving the accuracy and generalization ability of the model.

[0150] Specifically, the vector representation C obtained by the first-stage attention is taken as input, and self-matching calculation is performed on itself to obtain a self-matching representation. For each time step t, the weight score obtained is:

[0151]

[0152] In formula (1-12), c t is the state representation after the first-stage attention at time t; W S is a parameter matrix; is the attention score of the state vector at time t.

[0153] Then, the attention score a 2 calculated is multiplied by the state vector matrix C calculated by the first-stage attention to obtain the second-stage attention state representation:

[0154]

[0155] In formula (1-13), s t is the current state representation calculated by the attention weight at time t; is the final self-matching representation matrix.

[0156] Finally, the self-attention matching representation s t is combined with the original state representation c t in combination, to obtain a new self-matching enhanced representation f t t is a learnable gate vector representation determined by f t t to determine the final state representation

[0157] f t = tanh(W f [c t ,s t ]) (1-14)

[0158] g t = sigmoid(W g [c t ,s t ]) (1-15)

[0159]

[0160] In formulas (1-14) and (1-15) and (1-16), W f , W g are parameter matrices respectively.

[0161]

[0162] In a possible implementation, the process of predicting the question type and encoding by the question type encoder is as follows:

[0163] Although many neural network models have achieved good performance in the question generation task, there are still some phenomena that the generated answers do not match some question types. Therefore, in order to improve this phenomenon, by adding a question type encoder, the question type is predicted and integrated into the Seq2Seq model architecture to guide the generation of questions, so as to process the mismatched questions. Specifically, the question type prediction process is as follows:

[0164] ​​In the SQuAD1.1 large-scale question answering data set, about 100,000 (context-question-answer) pairs annotated by artificial are contained, and the content involves biography, history, geography, society, science, technology, mathematics and other fields. The invention counts the questions annotated by artificial in the data set, and the interrogative words of the questions are mainly "How", "Why", "Who", "Which", "Where", "When", "What", "Was", "Has", "Did". The remaining some other rare interrogative word types are all included in the "Other" category. Based on this, the existing questions in the SQuAD1.1 data set are labeled by engineering means to construct a question type classification data set. Some types of interrogative words with small quantities such as "Was", "Has", "Did" are supplemented from the NewsQA data set to ensure the balance of the sample quantity.

[0165] Then, the context information, the answer information, the answer named entity information and the answer part-of-speech information are taken as a group of training data, and an appropriate amount of data is taken for each category to balance the quantity of each category. Then, the prediction task is classified as a typical text classification task, and a fine-tuned BERT model is used as a classification model.

[0166] Firstly, each element in the context information, the answer information, the answer named entity information and the answer part-of-speech information of each input sample of the BERT model is converted into a corresponding token sequence, and the "SPE" symbol is used to separate each element. The "CLS" symbol is added before the input sequence to generate the vector representation of the sequence:

[0167] S=[ <cls>,context, <sep>, answer, <sep>,ner, <sep>pos] (1-18)

[0168] In formula (1-18), the context is the sentence information where the answer is located; the answer is the answer information; the ner is the named entity information of the answer; and the pos is the part-of-speech information of the answer. <sep>Separator.

[0169] Then use BERT to extract features from the sequence:

[0170] X hidden = BERT(S) (1-19)

[0171] Next, use the fully connected layer to extract features from the first position of the hidden layer (the feature vector corresponding to CLS). This feature vector includes all semantic information, and the fully connected layer will reduce the dimension of the CLS feature vector to the number of labels t, where t is the number of question type categories:

[0172] Y = Linear t (X hidde ) (1-20) Finally, use the Softmax function to calculate the corresponding label, and the final output result is the label y:

[0173] y = softmax(Y) (1-21) Regarding the encoder part of the question type encoder, the present application uses a Bi-LSTM model as the question type encoder for encoding. The encoder accepts two inputs, the answer a and the question type qt. Among them, the question type qt is obtained from the label y predicted by the question type prediction, and the embedding vector representation of the answer and the question type is x q :

[0174] x q = [a, qt] (1-22)

[0175] In formula (1-22), x q is an embedding vector; a is an answer vector feature; qt is a question type vector feature; and " is a vector concatenation operation in parallel mode.

[0176] Next, x q is passed to the Bi-LSTM layer to obtain the state vector representation at each time t:

[0177]

[0178] Finally, each time step t of the Bi-LSTM produces two hidden vector sequences, the forward sequence and the backward sequence, as shown in formula (1-24), and the hidden vector sequence at each t time is represented as:

[0179]

[0180] In formula (1-24) are represented as the hidden vector representation of the forward and backward sequences at t time, respectively; and h t The final hidden vector representation at time t obtained by the concatenation operation of the front and back sequences is represented as:

[0181]

[0182] H in formula (1-25) t The final hidden vector representation matrix L obtained represents the input length of the answer information and the question type.

[0183] In one possible implementation, the decoding process of the question decoder is as follows:

[0184] The output sequence of the multi-feature encoder is decoded by using a unidirectional LSTM model, and the information sequence generated by the multi-feature encoder is used as the input sequence to generate the corresponding target sequence step by step using the unidirectional LSTM decoder. Unlike the traditional decoding method, the last hidden layer state of the question type decoder is used as the initialization state u0 of the decoder. This initialization method helps to better guide the model to generate accurate interrogative words, thereby improving the problem of mismatch between question type and answer. At each step t of the decoder, the hidden layer state of the decoder is updated from the previous hidden layer state u t-1 and the previous predicted information y t-1 in the output sequence.

[0185] u t =LSTM(u t-1 ,y t-1 ) (1-26)

[0186] During the decoding process, the invention also uses an attention mechanism, which can effectively improve the model performance of the Seq2Seq encoder-decoder architecture. At each hidden layer state update, the attention score r t is calculated by re-considering all the hidden layer state vectors of the input sequence.

[0187] r t =H S T W d u t (1-27)

[0188] In formula (1-27), H S is the final state representation of the multi-feature encoder; W d is a parameter matrix; and u t is the hidden state representation of the decoder at time t.

[0189] Subsequently, the attention score is normalized by a softmax function to obtain the attention weight at each time step t:

[0190] a t = softmax(r t ) (1-28)

[0191] The final decoding state at time t is obtained by the original decoder hidden layer state vector u t and the attention weighted state vector c t

[0192] c t = H d ·a t (1-29)

[0193]

[0194] In the decoding stage, the copy mechanism based on the maximum output pointer and the pointer generation network method are adopted, and the final sentence result is obtained by calculating the probability of copying words from the input and the probability of generating words from the vocabulary.

[0195] The probability of generating words from the vocabulary in the generation process is:

[0196]

[0197] When calculating the copy probability, the possibility of word repetition in the input paragraph sequence is considered: some words may have a high frequency in the article, which means that their attention weights may be larger after summation than other words, thereby affecting the quality of the generated result, causing some unimportant words to obtain an excessively high weight. In order to solve this problem, the invention adopts a strategy based on the maximum output pointer: when a repeated word appears, only the maximum attention weight of the word is selected as the final weight distribution. Words that do not appear in the input sequence are set to negative infinity. As shown in equation (1-32):

[0198]

[0199] The probability P copy (w) of copying the original text in the generation process is:

[0200] P copy (w) = softmax(W v sc t ) (1-33) The final generation probability P gen (w) is represented as:

[0201] P gen (w) = P copy (w) + P vocab (w) (1-34)​

[0202] The following experiment verifies the two-stage attention-based multi-feature fusion problem generation method of the embodiment of the application through experiments.

[0203] The comparison experiment adopts multiple algorithm models, including typical algorithm models and relatively advanced algorithm models. These algorithm models are widely used in this field and are considered to be representative. Through these comparison experiments, the performance and effect of these algorithm models are evaluated to verify the effectiveness of the model method of the application. The following will list each algorithm model in detail:

[0204] 1) S2S+ATT: a seq2seq model with attention mechanism.

[0205] 2) NQG++: a seq2seq model with copy and attention mechanism, and answer position feature and word feature are added.

[0206] 3) ASs2s: an answer separation seq2seq, which better utilizes information from the paragraph and the target answer. By replacing the target answer in the original paragraph with a special token, the model learns to identify which interrogative word should be used.

[0207] 4) S2sa: a gated self-attention encoding and maximum output pointer mechanism in decoding to solve the challenge of processing long text input in question generation.

[0208] 5) CGC-QG: a multi-task learning framework that learns to use rich syntactic features to decide which words to generate and copy to guide the model to learn the accurate boundary between copying and generating.

[0209] 6) Multi-att: to link the extensive document context with the target answer, a multi-stage attention mechanism is used to represent the relevant context, which constitutes the basis of the sequence-to-sequence model to address the challenge of modeling long documents.

[0210] 7) G2S-Bert-RL: a Graph2Seq QG model based on reinforcement learning. The model includes a Graph2Seq generator and a hybrid assigner, where the Graph2Seq generator uses a bidirectional gated graph neural network-based encoder to embed the text, and the hybrid assigner has a hybrid objective that combines cross-entropy and RL loss to ensure that the generated text is syntactically and semantically valid.

[0211] 8) EGSS: an entity-guided question generation model based on the capture of contextual structure information and sequence information. This model uses a graph convolution network and a bidirectional long short-term memory network to simultaneously capture the structural information and sequence information of the context. In addition, in order to improve the answerability of the generated question, an entity-guided method is used to obtain the question type from the answer, and the answer and the question type are jointly encoded.

[0212] 9) CKFC: fine-grained division of question types, and integration of syntax knowledge into the question type quantifier to improve the accuracy of the question type. Then, the type predicted by the question type classifier is fed to the question generator.

[0213] To verify the effectiveness of the method, experiments were conducted on the SQuAD dataset version 1.1 Split1, and BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, METROT indicators were used as evaluation indicators, and the experimental results are shown in the table below.

[0214]

[0215] From the data of the comparative experimental results in the table, it can be analyzed that in the comparative experiment, the MT2SAtt model proposed by the method is compared with several classic question generation models and more advanced models, including paragraph-level models and sentence-level models: S2S+ATT, NQG++, ASs2s, S2sa, CGC-QG, Multi-att, G2S-Bert-RL, EGSS, etc. Among them, the model marked with "*" is a sentence-level task model. The experimental results show that the MT2SAtt model is better than these models in most evaluation indicators, especially in the BLEU-4 and ROUGE-L two evaluation indicators, which is obviously better than other models. This shows that the model of the application can better generate questions with correct grammar and rich semantics. Compared with the classic S2S+ATT model, the model of the application improves the performance by 53.24%. This result shows that the multi-feature fusion question generation method based on two-stage attention (MT2SAtt) has a significant improvement in processing paragraph-level text. Compared with other paragraph-level models such as Ass2s, S2sa, Multiatt, etc., the model of the application still has an advantage in performance. In addition, in the sentence-level question generation model, although some models such as CGC-QG, G2S-Bert-RL, EGSS, GKFC, etc. have achieved good results through multi-task learning, reinforcement learning and additional encoders, etc., there is still some gap in performance compared with the model of the application. And because the sentence-level model does not specially process the paragraph-level context, it will face the problem of noise information and long-distance dependence of text when processing the paragraph-level context, which will further reduce its performance.

[0216] The application has been verified that the multi-feature fusion question generation method based on two-stage attention mechanism (MT2SAtt) can obtain better generation performance by using a multi-feature encoder, a question type encoder and a question decoder, and introducing a two-stage attention mechanism.

[0217] Another embodiment of the application also provides a multi-feature fusion question generation system based on two-stage attention, comprising:

[0218] A feature vector embedding module is configured to input a context into a multi-feature encoder, embed a feature vector by the multi-feature encoder, and adopt two-stage attention encoding.

[0219] A question type prediction module is configured to input a context into a question type encoder, predict a question type and encode the question type.

[0220] A question decoding module is configured to input two encodings obtained by the multi-feature encoder and the question type encoder into a question decoder by adopting a cross-attention mechanism, and output a generated question by the question decoder.

[0221] The embodiment of the application also provides an electronic device, comprising a memory storing at least one instruction, and a processor executing the instruction stored in the memory to implement the multi-feature fusion question generation method based on two-stage attention.

[0222] The embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the multi-feature fusion question generation method based on two-stage attention.

[0223] For example, the instruction stored in the memory can be divided into one or more modules / units, which are stored in the computer readable storage medium and executed by the processor to complete the multi-feature fusion question generation method based on two-stage attention. The one or more modules / units can be a series of computer readable instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the server.

[0224] The electronic device can be a smart phone, a notebook, a palm computer, a cloud server, and the like. The electronic device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the electronic device can further include more or less components, or combine certain components, or different components, for example, the electronic device can further include an input / output device, a network access device, a bus, and the like.

[0225] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0226] The memory can be an internal storage unit of the server, such as a hard disk or a memory of the server. The memory can also be an external storage device of the server, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the memory can include both the internal storage unit and the external storage device of the server. The memory is used to store the computer readable instructions and other programs and data required by the server. The memory can also be used to temporarily store data that has been output or will be output.

[0227] It should be noted that the information interaction, execution process, and the like between the above module units, since based on the same concept as the method embodiments, the specific functions and the technical effects brought about can be referred to the method embodiment part, and will not be described here.

[0228] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the unit and module in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0229] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer-readable medium at least includes any entity or device that can carry a computer program code to a photographing device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc.

[0230] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0231] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.< / sep> < / sep> < / sep> < / sep> < / cls> < / sep> < / sep> < / sep> < / sep> < / cls>

Claims

1. A two-stage attention-based multi-feature fusion question generation method, characterized in that, The method comprises the following steps: inputting the context into a multi-feature encoder, embedding a feature vector by the multi-feature encoder, and adopting two-stage attention encoding; inputting the context into a question type encoder, predicting a question type and encoding; adopting a cross-attention mechanism to fuse the two encodings obtained by the multi-feature encoder and the question type encoder, and inputting the two encodings into a question decoder, and outputting a generated question by the question decoder; the multi-feature encoder adopts a Bi-LSTM model for encoding, and is composed of two LSTM layers based on an RNN structure, one of which processes an input sequence from the beginning to the end, and the other of which processes an input sequence from the end to the beginning; the outputs of the two directions are connected at each time to form a final output of the Bi-LSTM model; the first-stage attention text in the two-stage attention encoding adopts a soft attention matching matrix; the second-stage attention mechanism in the two-stage attention encoding adopts a gated self-attention fusion method; the question decoder adopts a unidirectional LSTM model to decode the output sequence of the multi-feature encoder, and uses a unidirectional LSTM decoder to generate a corresponding target sequence step by step by taking the information sequence generated by the multi-feature encoder as an input sequence. 2.The two-stage attention based multi-feature fusion question generation method of claim 1, wherein, The multi-feature encoder, the question type encoder and the decoder are established based on the encoding-decoding architecture of a Seq2Seq model. 3.The two-stage attention based multi-feature fusion question generation method of claim 1, wherein, In the step of embedding the feature vector by the multi-feature encoder, the words in the context sentence are converted into corresponding pre-trained word vectors, and the pre-trained word vectors are input into the multi-feature encoder, while considering language features and sentiment features, semantic role labeling features and dependency syntax features, the language features including named entities, parts of speech and case features.

4. The method of claim 1, wherein, The input of the multi-feature encoder includes word embedding, answer label features, part of speech features, named entity features and answer sentence features; wherein the word embedding adopts a pre-trained GloVe embedding representation, and uses global co-occurrence statistical information to learn a word vector, and each word in the text is given a corresponding word embedding vector feature by GloVe; the answer label features: each word in the paragraph text is labeled, and a BIO label method is adopted, the label B represents the beginning of the answer, the label I represents a continuous part of the answer, and the label O marks a word that does not constitute a part of the answer; the part of speech feature: refers to an attribute used to describe the grammatical function of a word in a natural language processing task, the part of speech feature is One-Hot one-hot encoded, and then the feature is extracted by an encoder; the named entity feature: refers to an attribute used to describe a word or phrase with a specific entity name in a natural language processing task, and the named entity is also One-Hot one-hot encoded; the answer sentence feature: describes a sentence containing a correct answer in the text, if a sentence in the paragraph contains an answer fragment, all the words in the corresponding sentence are marked as 1, and the words in other sentences that do not contain the answer are marked as 0.

5. The method of claim 1, wherein, The input received by the Bi-LSTM model of the multi-feature encoder includes an answer a, a paragraph P to which the answer belongs, and a plurality of feature combinations mentioned above. The answer a, the paragraph P to which the answer belongs, the embedding vector obtained by the answer are respectively , : =[ ] =[ ] In the formula, is a word vector feature; is a named entity vector feature; is a part-of-speech feature vector; is a sentence feature where the answer is located; is an answer token vector feature; a concatenation operation in series; an embedded representation of the answer; an embedded representation of the paragraph P to which the answer belongs; i Indicates the first in the text i One word; The answer a, the embedding vector of the paragraph P to which the answer belongs , are respectively passed into a shared Bi-LSTM layer, and the state vector representation of each time t is obtained after encoding , : =Bi-LSTM ( ) =Bi-LSTM ( ) wherein is the state at the previous time instant; Each time through the Bi-LSTM layer produces two sequences of hidden vectors, a forward sequence and a backward sequence, so that each time t t, there are two hidden vectors, ftand bt, representing the forward and backward sequences, respectively. t :​ = [ ; ] wherein, , are the forward and backward sequence hidden vector representations at each time t instance, respectively; are the final hidden vector representations at each time t instance, obtained by concatenating the forward and backward sequence hidden vector representations. = = In the formula, , is the final resulting state representation matrix. 6.The two-stage attention based multi-feature fusion question generation method of claim 5, wherein, The first stage attention text adopts a soft attention matching matrix, including obtaining the attention matrix of the paragraph context to the answer by calculating the mutual connection between the answer representation and the context representation: = wherein, is the word vector representation of the i th word in the passage; is the word vector representation of the j th word in the answer; is a resulting element in the attention matrix; = wherein (·) is a standard non-linear transformation function, is a Sigmoid function; is a parameter matrix; Adopting matrix M The maximum pooling operation in the column direction generates an attention vector: = wherein, is the attention weight score obtained; is the soft attention matching matrix; each element ∈ is the relevance of the i-th word in passage P to answer A captured based on the maximum relevance of the i-th word in passage P to answer A; i to answer A. applying attention scores to , resulting in a passage representation with associated weight coefficients : = wherein denotes element-wise multiplication, is a vector representation of attention through the first stage, where words directly related to the answer are amplified when the attention score is high, and irrelevant words are filtered out when the attention score is low.

7. The method of claim 6, wherein, The vector representation from the first stage attention is As input, and computes a self-matching representation from the attention matching on the input itself, each time t The attention score for the state vector at time Is computed as follows: = wherein is the state representation at each time t t, and is a parameter matrix; the computed attention scores are multiplied with the vector representation obtained from the one-stage attention to obtain the two-stage attention post-state representation: = wherein is the current state representation computed at time t time through the attention weight computation; S = is the resulting self-matching representation matrix; The current state representation at each time t is computed by attention weight The current state representation at each time t is combined with the state representation of the previous time step through a one-stage attention , The final state representation is determined by a learnable gate vector , The final state representation is determined by a learnable gate vector The computation expression is as follows: = = = + wherein , are parameter matrices, respectively. = 。 8.The two-stage attention based multi-feature fusion question generation method of claim 1, wherein, The question type encoder counts the questions labeled by artificial in the SQuAD1.1 question answering dataset, the interrogative words of the questions include How, Why, Who, Which, Where, When, What, Was, Has and Did, and other interrogative word types are included in the Other category, and the interrogative word types with small quantities are supplemented from the NewsQA dataset; The context information, the answer information, the answer named entity information and the answer part-of-speech information are taken as a group of training data, the number of each category is balanced, the prediction task is classified as a typical text classification task, and a fine-tuned BERT model is used as a classification model; For each element in the context information, the answer information, the answer named entity information and the answer part-of-speech information of each input sample of the BERT model, it is converted into a corresponding token sequence, and the SPE symbols are separated; the symbol is added before the input sequence CLS sequence to generate a vector representation of the sequence: = In the formula, context is the answer sentence information; answer is the answer information; ner is the named entity information of the answer; pos is the part-of-speech information of the answer; is the delimiter; Then the BERT model is used to extract features of the sequence: =BERT Using a fully connected layer on the first position of the hidden layer CLS The corresponding feature vector extracts features, the feature vector The fully connected layer will include all semantic information CLS The feature vector dimension will be the number of label t , t The size of the problem type class number: = Using Softmax The function calculates the corresponding label, and the final output result is the label y : y= The question type encoder encodes with a Bi-LSTM model to answer a and question type qt as input, question type qt label predicted by question type y The resulting, answer and question type resulting embedding vector representations are ​ = wherein is an embedding vector; a is an answer vector feature; qt is a question type vector feature; notation ， denotes a vector concatenation operation in parallel fashion; passed to the Bi-LSTM layer, which encodes to get the state vector representation of each time t moment : = Two hidden vector sequences, a forward sequence and a backward sequence, are generated at each time step t The hidden vector sequences at each time step are represented as: t The hidden vector sequences at each time step are represented as: = wherein are represented as final hidden vector representations for each time t are represented as final hidden vector representations for each time are represented as final hidden vector representations for each time t are represented as final hidden vector representations for each time = In the formula, denotes the final obtained hidden vector representation matrix, L is the input length of the answer information and question type. 9.The two-stage attention based multi-feature fusion question generation method of claim 1, wherein, Using the last hidden layer state of the problem type decoder as the initialization state of the decoder ; at each time t instance of the decoder, the hidden layer state is derived from the previous hidden layer state and the previous prediction information in the output sequence results in: = LSTM The attention scores are computed by re-considering all the hidden state vectors of the input sequence each time the hidden state of a layer is updated : = wherein is the final state representation of the multi-feature encoder; is the parameter matrix; is the decoder t is the hidden state representation at time The attention score is normalized by a softmax function to obtain t Attention weight at time t: =softmax By t the original decoder hidden layer state vector and the attention weighted state vector the final t decoding state : = · =tanh In the decoding stage, a copying mechanism based on the maximum output pointer and a pointer generation network method are used, and by calculating the probability of copying words from the input and the probability of generating words from the vocabulary, the final sentence result is obtained; The probability of generating words from the vocabulary in the generation process is: =softmax The strategy based on the maximum output pointer is adopted, when a repeated word appears, only the maximum attention weight corresponding to the word is selected as the final weight distribution, and the words not appearing in the input sequence are set to negative infinity: = Probability of copying original text in the generation process is: =softmax Final generation probability is represented as: = + 。 10. A two-stage attention based multi-feature fusion question generation system for implementing the two-stage attention based multi-feature fusion question generation method according to any one of claims 1 to 9, characterized in that, It includes: The feature vector embedding module is used for embedding the context input into the multi-feature encoder, and the feature vector is embedded by the multi-feature encoder, and two-stage attention encoding is adopted; The question type prediction module is used for inputting the context into the question type encoder to predict the question type and encode; The question decoding module is used for inputting the two encodings obtained by the multi-feature encoder and the question type encoder into the question decoder by using the cross attention mechanism, and the question generated by the question decoder is output.

Images