A question and answer matching method based on adaptive fusion multi-attention network

Through the combination of multiple self-attention networks and sequence attention networks, adaptively integrate information, the problem of limited information collection ability in the existing technology is solved, and information extraction and matching is achieved from multiple angles and aspects, which significantly improves the answers Accuracy of choice.

CN112966499BInactive Publication Date: 2024-12-03SUN YAT SEN UNIV
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202110287825.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-17
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing multi-self attention network model enhances the understanding of questions and answers through multiple rounds of processing, but fails to obtain multi-angle and multi-faceted information of questions and answers, resulting in limited information collection capabilities.

Method used

Multiple self-attention networks are used to extract information from different aspects of the problem, and generate corresponding answer vectors through the serial attention network, calculate the matching score, adaptively fuse the matching scores of multiple aspects of information, and match the questions and answers. .

Benefits of technology

It significantly improves network performance, can extract information about questions and answers from multiple angles and aspects, and improves the accuracy of answer selection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112966499B_ABST
    Figure CN112966499B_ABST
Patent Text Reader

Abstract

The present invention provides a method for matching questions and answers based on an adaptive fusion multi-attention network, comprising the following steps: S1: converting each word in a question and an answer into a word vector respectively, and encoding the word vector to obtain a word vector sequence; S2: using multiple self-attention networks to extract information of different aspects in a question, and respectively encoding them into different question vectors; S3: generating a corresponding answer vector for each question vector; S4: calculating a matching score, and adaptively fusing the matching scores of multiple aspects of information, and matching questions and answers according to the fused matching scores. The present invention provides a method for matching questions and answers based on an adaptive fusion multi-attention network, which solves the problem that the current model using multiple self-attention networks enhances the understanding of questions and answers through multiple rounds of processing, rather than obtaining multi-angle and multi-faceted information of questions and answers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more specifically, to a question and answer matching method based on an adaptive fusion multi-attention network. Background Art

[0002] Deep learning has a strong function fitting ability. Using deep learning for answer selection tasks has the advantages of fast running speed, easy calculation, and better than traditional effects. By carefully designing the network structure and simulating the thinking process of people when choosing answers, it is expected that good results can be achieved. From previous studies, we know that the learning of question representation plays a very important role in answer selection. A good answer selection model should be able to generate high-quality question vectors and answer vectors, and comprehensively capture the interactive relationship between questions and answers. In fact, it is difficult for a simple self-attention network to collect useful information for all questions because the fitting ability of a two-layer forward network is limited. In addition, due to its timing characteristics, it is more difficult to train than a general fully connected network. Obviously, it is difficult to have the ability to collect information applicable to all questions. One method is to use multiple self-attention networks and then force each self-attention network to pay attention to different parts of the sentence. It introduces a penalty term during training, so that the more similar the weights generated by different self-attention networks are, the greater the additional penalty loss is, thereby achieving the purpose of forcing attention to different parts. However, the current model using multiple self-attention networks is performed through multiple rounds of processing, and there are dependencies between each round of processing, so it should be understood more as enhancing the understanding of questions and answers through multiple rounds of processing, rather than obtaining multi-angle and multi-faceted information about questions and answers.

[0003] In the prior art, such as the Chinese patent published on November 23, 2018, the answer selection method, device and electronic device based on the cross-attention neural network, with the publication number CN108875074A, the mutual information between different candidate answers is cross-scored through the attention mechanism to determine the correlation between the question data and the answer data, thereby improving the accuracy of the answer selection, but does not obtain multi-angle and multi-faceted information of the questions and answers. Summary of the invention

[0004] In order to overcome the technical defect that the current model using multiple self-attention networks enhances the understanding of questions and answers through multiple rounds of processing instead of obtaining multi-angle and multi-faceted information of questions and answers, the present invention provides a question and answer matching method based on an adaptive fusion multi-attention network.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A question and answer matching method based on adaptive fusion multi-attention network, comprising the following steps:

[0007] S1: Convert each word in the question and answer into a word vector respectively, and encode the word vectors in the question and answer to obtain the word vector sequence of the question and answer;

[0008] S2: Use multiple self-attention networks to extract different aspects of a question based on the word vector sequence and encode them into different question vectors respectively;

[0009] S3: Use the sequence attention network to generate the corresponding answer vector for each question vector;

[0010] S4: Calculate the matching score of each question vector and its corresponding answer vector, and adaptively fuse the matching scores of multiple aspects of information by evaluating the weight of each question vector in a question, and match the question and answer according to the fused matching scores.

[0011] Preferably, in step S1, a bidirectional LSTM is used to encode the word vectors in the question and the answer respectively.

[0012] Preferably, given a sentence S containing l words, S = (w1, w2, ..., w l ), use the bidirectional LSTM to encode the word vector and obtain the corresponding hidden layer:

[0013]

[0014]

[0015]

[0016] When the given sentence S is a question, the hidden layer vector sequence H of each word vector in the question is obtained q ={h q (1),...,h q (l)}, H q ={h q (1),...,h q (l)} as the word vector sequence of the question;

[0017] When the given sentence S is the answer, the hidden layer vector sequence H of each word vector in the answer is obtained a ={h a (1),...,h a (l)}, H a ={h a (1),...,h a (l)} as the word vector sequence of the answer;

[0018] Among them, w1,w2,...,w l are l words in sentence S respectively, is the hidden state of the forward LSTM at time t, represents the encoding process of the forward LSTM, is the hidden state of the forward LSTM at time t-1, w t is the word vector encoded by LSTM at time t, is the hidden state of the backward LSTM at time t, represents the encoding process of backward LSTM encoding, is the state of the backward LSTM at time t-1, h t is the final encoding result of the bidirectional LSTM, h q (1),...,h q (l) are the l word vectors in the question, h a (1),...,h a (l) are the l word vectors in the answer.

[0019] Preferably, in step S2, the kth self-attention network calculates the question vector by the following formula: And each different question vector extracts information about different aspects of the problem:

[0020]

[0021]

[0022]

[0023] in, For the kth self-attention network, from the original question vector h q The attention vector extracted from (t) is and are the parameters of the kth self-attention network, is the weight of the word vector.

[0024] Preferably, step S3 comprises the following steps:

[0025] S3.1: Get the hidden layer vector sequence H of each word vector in the answer a ={h a (1),...,h a (l)};

[0026] S3.2: For the kth question vector By applying the same sequence attention model to Ha Generate the weight of the corresponding k-th answer word vector

[0027] S3.3: Calculate H a The weighted sum of

[0028] Preferably, in the sequence attention model, LSTM is used to calculate the weight of each answer word vector.

[0029] Preferably, the input of LSTM is defined as:

[0030]

[0031] Vector Input into a bidirectional LSTM to obtain the corresponding hidden layer

[0032]

[0033] Then the hidden layer Add up all dimensions of to get the weight score; then get the weight of the answer word vector

[0034]

[0035] Finally, the answer vector is obtained by calculating the weighted sum of the answer hidden layer

[0036]

[0037] Preferably, in step S4, the matching score of each question vector and its corresponding answer vector is calculated by cosine similarity.

[0038] Preferably, the weight of each question vector in a question is evaluated by using a multi-layer perceptron network.

[0039] Preferably, the matching scores of multiple aspects of information are adaptively integrated through the following formula:

[0040]

[0041]

[0042]

[0043] Among them, q represents the question, a represents the answer, K is the total number of question vectors, and α k is the weight of the kth question vector, is the kth question vector, is the kth answer vector, W s1 and are all trainable network parameters, s k is the attention vector obtained by self-attention for the kth question vector.

[0044] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0045] The present invention provides a question and answer matching method based on an adaptive fusion multi-attention network, which utilizes multiple self-attention networks to focus on different directions, thereby extracting information on questions and answers from multiple angles and aspects, and adaptively fusion the extracted information, greatly improving network performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of the implementation steps of the technical solution of the present invention;

[0047] Figure 2 It is a schematic diagram of the implementation framework of the technical solution of the present invention. DETAILED DESCRIPTION

[0048] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0049] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0050] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0051] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0052] Example 1

[0053] like Figure 1 As shown, a question and answer matching method based on adaptive fusion multi-attention network includes the following steps:

[0054] S1: Convert each word in the question and answer into a word vector respectively, and encode the word vectors in the question and answer to obtain the word vector sequence of the question and answer;

[0055] S2: Use multiple self-attention networks to extract different aspects of a question based on the word vector sequence, and encode them into different question vectors respectively; one self-attention network extracts one aspect of a question, and a corresponding question vector is obtained;

[0056] S3: Use the sequence attention network to generate the corresponding answer vector for each question vector;

[0057] S4: Calculate the matching score of each question vector and its corresponding answer vector, and adaptively fuse the matching scores of multiple aspects of information by evaluating the weight of each question vector in a question, and match the question and answer according to the fused matching scores.

[0058] Example 2

[0059] More specifically, Figure 2 As shown, in step S1, a bidirectional LSTM is used to encode the word vectors in the question and the answer respectively, so as to obtain a word vector representation containing context information.

[0060] More specifically, given a sentence S = (w1, w2, ..., w l ), use the bidirectional LSTM to encode the word vector and obtain the corresponding hidden layer:

[0061]

[0062]

[0063]

[0064] When the given sentence S is a question, the hidden layer vector sequence H of each word vector in the question is obtained q ={h q (1),...,h q (l)}, H q ={h q (1),...,h q (l)} as the word vector sequence of the question;

[0065] When the given sentence S is the answer, the hidden layer vector sequence H of each word vector in the answer is obtained a ={h a (1),...,h a (l)}, H a ={h a (1),...,h a (l)} as the word vector sequence of the answer;

[0066] Among them, w1,w2,...,w l are l words in sentence S respectively, is the hidden state of the forward LSTM at time t, represents the encoding process of the forward LSTM, is the hidden state of the forward LSTM at time t-1, w t is the word vector encoded by LSTM at time t, is the hidden state of the backward LSTM at time t, represents the encoding process of backward LSTM encoding, is the state of the backward LSTM at time t-1, h t is the final encoding result of the bidirectional LSTM, h q (1),...,h q (l) are the l word vectors in the question, h a (1),...,h a (l) are the l word vectors in the answer.

[0067] More specifically, in step S2, the kth self-attention network calculates the question vector by the following formula: And each different question vector extracts information about different aspects of the problem:

[0068]

[0069]

[0070]

[0071] in, For the kth self-attention network, from the original question vector h q (t) The attention vector h is extracted from q (t) is obtained from the final hidden state of the bidirectional LSTM, and are the parameters of the kth self-attention network, is the weight of the word vector.

[0072] More specifically, step S3 includes the following steps:

[0073] S3.1: Get the hidden layer vector sequence H of each word vector in the answer a ={h a (1),...,h a (l)};

[0074] S3.2: For the kth question vector By applying the same sequence attention model to H a Generate the weight of the corresponding k-th answer word vector

[0075] S3.3: Calculate H a The weighted sum of

[0076] More specifically, in the sequence attention model, LSTM is used to calculate the weight of each answer word vector.

[0077] In the specific implementation process, LSTM is used to calculate the weight of each answer word vector, instead of using perceptron or bilinear product as in the general attention model. In this way, contextual information can be introduced when calculating the weight.

[0078] More specifically, the input of LSTM is defined as:

[0079]

[0080] Different from the original setting of the sequence attention network, and h a The element-wise multiplication and Euclidean distance of (t) are used as input; the significance of element-wise multiplication is to obtain and the hidden layer h for each word a The significance of Euclidean distance is to obtain the differences between various dimensions; thus, more accurate sequence attention can be obtained, which improves the accuracy of screening;

[0081] Vector Input into a bidirectional LSTM to obtain the corresponding hidden layer

[0082]

[0083] Then the hidden layer Add up all dimensions of to get the weight score; then get the weight of the answer word vector

[0084]

[0085] Finally, the answer vector is obtained by calculating the weighted sum of the answer hidden layer

[0086]

[0087] More specifically, in step S4, the matching score of each question vector and its corresponding answer vector is calculated by cosine similarity.

[0088] More specifically, a multi-layer perceptron network is used to evaluate the weight of each question vector in a question.

[0089] More specifically, the matching score of multiple aspects of information is adaptively integrated through the following formula:

[0090]

[0091]

[0092]

[0093] Among them, q represents the question, a represents the answer, K is the total number of question vectors, and α k is the weight of the kth question vector, is the kth question vector, is the kth answer vector, W s1 and are all trainable network parameters, s k is the attention vector obtained by self-attention for the kth question vector.

[0094] In the specific implementation process, although the question generates k question vectors, in fact not all question vectors are meaningful, because the question generally does not have so much information. Therefore, when a question vector does extract meaningful information, it is necessary to give a high weight to the matching score of this question vector. When it does not extract information, it needs to be given a low weight.

[0095] Example 3

[0096] In this embodiment, three public data sets cited by multiple papers are selected as evaluation data sets to conduct an empirical study on the question and answer matching method based on an adaptive fusion multi-attention network.

[0097] Table 1 shows the situation of three public datasets.

[0098] Table 1

[0099] Dataset TrecQA WikiQA InsuranceQA Number of training / validation / test questions 1162 / 65 / 68 873 / 126 / 243 12887 / 1000 / 1800x2 Average question length 8 6 7 Average answer length 28 25 95 Average number of candidate answers 38 9 500

[0100] In order to make a fair comparison with previous studies, this example uses the evaluation criteria of previous researchers on the same dataset. On TrecQA and WikiQA, this example uses MeanAveragePrecision (MAP) and MeanReciprocalRank (MRR) to evaluate the performance of the model. On the other hand, InsuranceQA uses the top1 accuracy rate used by previous researchers, which requires that the correct answer must be ranked first. For all compared models, this example reports the accuracy rates in their original papers.

[0101] The question and answer matching method based on the adaptive fusion multi-attention network (AFMAN) is implemented using the pytorch framework, using a 300-dimensional Glove pre-trained with 840B text as a word vector. The word vector remains unchanged during the training process. The bidirectional LSTM that encodes the question and answer shares parameters. The LSTM that encodes the sentence and the sequence attention is composed of one layer. In order to prevent overfitting, this embodiment uses a dropout layer with p=0.3, which is set before the input of the LSTM. During the training process, this embodiment calculates the marginloss (margin=0.1) of each negative sample and positive sample of each question. In order to further improve the effect, this embodiment calculates the loss of multiple negative samples for each question, but only takes the negative sample with the largest loss for optimization. In addition, the average word vector is used as an additional question vector to improve performance. For all data sets, this embodiment selects the Adam optimizer. During the training process, this embodiment adjusts the parameters on the validation set, and uses the model with the best effect on the validation set to observe the effect on the test set.

[0102] For the TrecQA dataset, the batch size is set to 100, and the hidden layer of the LSTM responsible for encoding sentences is set to 141. In addition, in the self-attention network and the sequence attention network, the hidden layer is set to 282. In this dataset, the number of self-attention networks K is set to 5, and the training is iterated 20 times on the training set. When training the InsuranceQA dataset, similar parameter settings to TrecQA are used, except that the number of training iterations is set to 50 (because the dataset is larger). WikiQA is the smallest dataset tested in this embodiment. Therefore, this embodiment sets the hidden layer of LSTM to 96 and the hidden layer of the attention network to 144. In addition, since more self-attention networks do not help performance in wikiQA, K is set to 1 to speed up the operation. During training, the batch size is set to 50 and the training is iterated 20 times.

[0103] Table 2 shows the experimental results on the TrecQA dataset.

[0104] Table 2

[0105] Model MAP MRR QA-LSTM 0.682 0.765 QA-LSTM / CNN with 0.728 0.824 AP-BILSTM 0.713 0.803 AP-CNN 0.753 0.851 IARNN-Gate 0.737 0.821 RNN-POA 0.781 0.851 BiMPM 0.802 0.875 Multihop Attention Networks 0.813 0.893 AFMAN 0.827 0.902

[0106] The accuracy on the TrecQA dataset is shown in Table 2. AFMAN in this embodiment not only significantly surpasses the classic QA-LSTM model, but also performs better than the recently proposed attention model. Compared with AP-CNN, which uses a two-way attention mechanism, the attention method used by AFMAN has been experimentally proven to be a superior solution, surpassing the AP-CNN model by 7.4% and 5.1% in MAP and MRR, respectively. It also improves MAP / MRR by about 5% over the position-based attention RNN. Moreover, AFMAN performs better than the newly proposed BiMPM and multi-hop attention network, achieving advanced performance on TrecQA.

[0107] Table 3 shows the experimental results on the InsuranceQA dataset.

[0108] Table 3

[0109] Model MAP MRR QA-LSTM 0.631 0.580 QA-LSTM with attention 0.681 0.622 AP-CNN 0.698 0.663 AP-BILSTM 0.717 0.664 IARNN-Gate 0.701 0.628 Multihop Attention Networks 0.705 0.669 AFMAN 0.738 0.694

[0110] As can be seen from Table 1, insuranceQA is a larger dataset. Not only does it have more questions, but the average length of the answers is also much longer than TrecQA and WikiQA. This means that the answers may contain more information (including more redundant information), and the connection between questions and answers will be more complex, making the answer selection task more challenging. In Table 3, AFMAN significantly surpasses all other comparison methods, which means that AFMAN has superior performance in solving complex answer selection tasks.

[0111] Table 4 shows the experimental results on the WikiQA dataset.

[0112] Table 4

[0113]

[0114]

[0115] Table 4 reports the experimental results on the WikiQA dataset. First, AFMAN in this embodiment outperforms AP-CNN by about 4%. In addition, IARNN achieves the best MAP result, slightly surpassing AFMAN. Compared with the recent BiMPM and MAN, AFMAN has about 1-2% improvement in MAP and MRR. The improvement of AFMAN on WikiQA is not as obvious as that on InsuranceQA and TrecQA, which is attributed to the fact that the scale of WikiQA is much smaller than the other two datasets, so AFMAN with higher model complexity lacks sufficient training data for fitting. Overall, AFMAN also achieves good performance on the WikiQA dataset.

[0116] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A question and answer matching method based on adaptive fusion multi-attention network, characterized in that: The following steps are involved: S1: Convert each word in the question and answer into a word vector respectively, and encode the word vectors in the question and answer to obtain the word vector sequence of the question and answer; In step S1, a bidirectional LSTM is used to encode the word vectors in the question and answer respectively; Given a sentence S containing l words, S = (w1, w2, ..., w l ), use the bidirectional LSTM to encode the word vector and obtain the corresponding hidden layer: When the given sentence S is a question, the hidden layer vector sequence H of each word vector in the question is obtained q ={h q (1),...,h q (l)}, H q ={h q (1),...,h q (l)} as the word vector sequence of the question; When the given sentence S is the answer, the hidden layer vector sequence H of each word vector in the answer is obtained a ={h a (1),...,h a (l)}, H a ={h a (1),...,h a (l)} as the word vector sequence of the answer; Among them, w1,w2,...,w l are l words in sentence S respectively, is the hidden state of the forward LSTM at time t, represents the encoding process of the forward LSTM, is the hidden state of the forward LSTM at time t-1, w t is the word vector encoded by LSTM at time t, is the hidden state of the backward LSTM at time t, represents the encoding process of backward LSTM encoding, is the state of the backward LSTM at time t-1, h t is the final encoding result of the bidirectional LSTM, h q (1),...,h q (l) are the l word vectors in the question, h a (1),...,h a (l) are the l word vectors in the answer; S2: Use multiple self-attention networks to extract different aspects of a question based on the word vector sequence and encode them into different question vectors respectively; in step S2, the kth self-attention network calculates the question vector by the following formula And each different question vector extracts information about different aspects of the problem: in, For the kth self-attention network, from the original question vector h q The attention vector extracted from (t) is and are the parameters of the kth self-attention network, is the weight of the word vector; S3: Use the sequence attention network to generate the corresponding answer vector for each question vector; S4: Calculate the matching score of each question vector and its corresponding answer vector, and adaptively fuse the matching scores of multiple aspects of information by evaluating the weight of each question vector in a question, and match the question and answer according to the fused matching scores.

2. A question and answer matching method based on adaptive fusion multi-attention network according to claim 1, characterized in that: Step S3 includes the following steps: S3.1: Get the hidden layer vector sequence H of each word vector in the answer a ={h a (1),...,h a (l)}; S3.2: For the kth question vector By applying the same sequence attention model to H a Generate the weight of the corresponding k-th answer word vector S3.3: Calculate H a The weighted sum of 3. A question and answer matching method based on adaptive fusion multi-attention network according to claim 2, characterized in that: In the sequential attention model, LSTM is used to calculate the weight of each answer word vector.

4. A question and answer matching method based on adaptive fusion multi-attention network according to claim 3, characterized in that: The input of LSTM is defined as: Vector Input into a bidirectional LSTM to obtain the corresponding hidden layer Then the hidden layer Add up all dimensions of to get the weight score; then get the weight of the answer word vector Finally, the answer vector is obtained by calculating the weighted sum of the answer hidden layer 5. A question and answer matching method based on adaptive fusion multi-attention network according to claim 4, characterized in that: In step S4, the matching score of each question vector and its corresponding answer vector is calculated by cosine similarity.

6. A question and answer matching method based on adaptive fusion multi-attention network according to claim 5, characterized in that: The weight of each question vector in a question is evaluated by using a multi-layer perceptron network.

7. A question and answer matching method based on adaptive fusion multi-attention network according to claim 6, characterized in that: The matching score of multiple aspects of information is adaptively integrated through the following formula: Among them, q represents the question, a represents the answer, K is the total number of question vectors, and α k is the weight of the kth question vector, is the kth question vector, is the kth answer vector, W s1 and are all trainable network parameters, s k is the attention vector obtained by self-attention for the kth question vector.

Citation Information

Patent Citations

  • Method and device of answer selection based on cross-attention neural network and electronic equipment

    CN108875074A

  • Machine reading understanding method based on a multi-head attention mechanism and dynamic iteration

    CN109492227A

  • Machine reading comprehension answer obtaining method based on multi-round attention mechanism

    CN110083682A

  • Implementation method for fusing network question and answer system based on multi-attention mechanism

    CN110134771A

  • Machine reading understanding method and device, electronic equipment and computer storage medium

    CN112183085A