Deep learning text matching method based on grammatical information fusion in an enhanced way

By integrating grammatical information and Euclidean distance enhancement interaction process in the deep learning text matching model, the problem of existing models ignoring syntactic information is solved, and more accurate sentence pair matching and performance improvement is achieved.

CN115034202BActive Publication Date: 2024-11-01TIANJIN UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210384575.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-13
Publication Date
2024-11-01
Estimated Expiration
2042-04-13

AI Technical Summary

Technical Problem

The existing deep learning text matching model ignores the relative position relationship between syntactic information and word level in semantic modeling between sentences, resulting in the inclusion of semantic and syntactic information that is not detailed enough and cannot effectively capture the fine-grained interactive information between sentence pairs.

Method used

Deep learning text matching method that fuses grammatical information based on enhancement method is adopted, and dependent syntax tree is constructed through dependency syntax analysis, combined with deep neural network to extract dependent syntax information and semantic information of sentences, and use Euro-style distance to enhance internal product interaction when calculating similarity Process, information pooling feature extraction is performed by combining the pooling prediction layer.

Benefits of technology

It can better capture the semantic similarity and fine-grained syntactic position information between sentence pairs, improve the accuracy of sentence pair matching, reduce the amount of parameters, and improve the feasibility and performance of the model deployment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115034202B_ABST
    Figure CN115034202B_ABST
Patent Text Reader

Abstract

The present invention discloses a deep learning text matching method based on the fusion of grammatical information in an enhanced manner. The text matching method is established on a coding layer, a matching fusion layer and a pooling prediction layer; the coding layer is used to perform dependency syntactic analysis on sentences when encoding sentence pairs, construct a dependency syntactic tree, extract the dependency syntactic information and semantic information of the sentences by using a deep neural network, and add and fuse the two; the matching fusion layer is used to calculate the Euclidean distance of the encoding features between sentence pairs when calculating similarity, subtract the corresponding positions of the inner product interaction matrix and the Euclidean distance matrix, and enhance the inner product interaction process by using the Euclidean distance; the pooling prediction layer is used to extract the pooling features of information by using a pooling module that is a combination of a weighted pooling method and a maximum pooling method. The present invention has verified the ability to accurately match sentence pairs on both SNLI and Quora data sets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of text matching; in particular, it relates to a method for end-to-end question-answer matching by fusing grammatical information in an enhanced manner. Background Art

[0002] As deep learning has achieved very remarkable results on large-scale text interpretation and annotation datasets, a large number of deep learning matching models have emerged, which can be summarized into two matching architectures. One is a representational matching framework based on sentence encoding, and the other is an interactive matching architecture based on attention mechanism. For the first architecture, the overall semantic representation of sentence pairs is extracted through the encoding representation of the sentences for classification. The overall approach is simple and effective, but lacks fine-grained semantic interaction modeling between the two sentences. For the second framework, the interactive attention mechanism is introduced on the basis of the first framework to perform semantic matching modeling on word-level information, which can achieve better results.

[0003] However, on the one hand, in the existing deep learning matching models, more emphasis is placed on the modeling of sentence pair semantics, while ignoring the syntactic information of the sentence and the natural relative position relationship information between words in the sentence formed by syntax. Even if there are some models that use syntactic information, they only simply model the syntax, and the fusion of semantic and syntactic information features is relatively rough, and the balance between the importance of semantics and syntax cannot be grasped. For example, in the two examples given in Table 1, in the first example, the position exchange of "fish" and "eggs" does not affect the semantics of the sentence, because the two are connected by the coordinating conjunction "and", and "and" carries important local syntactic information. For the second example, although "US" and "Canada" are both place name entities, their positions in the sentence cannot be exchanged, because "from" and "to" are directional and carry the syntactic information of the master and the slave. It can be seen that modeling the dependent syntactic position information in the sentence is crucial. In recent years, some previous works have also proved the importance of syntactic position information in sentence pair matching modeling. On the other hand, the inner product matching method can only measure the similarity between sentence pairs by whether the directions of the vectors are similar, and cannot capture the interaction between the relative position information and syntactic information at the word level in the sentence pairs. The semantic similarity of the sentence pairs is reflected in the direction of the feature vectors, while the syntactic information and relative position information in the sentence pairs are reflected in each value of the feature vectors.

[0004] With the continuous development of deep learning and the widespread application of attention, some works in recent years are mostly based on interaction. For example, the ABCNN[1] model adds attention interaction on the basis of BCNN, and BiMPM[2] improves the interactive matching method from multiple different angles to better match sentence pairs. ESIM[3] is a very simple and efficient semantic matching model. In order to better capture the semantic interaction information between sentence pairs, more and more models have begun to emerge (MwAN[3], CAFE[4], CSRAN[5], DIIN[6], etc.). Among them, AF-DMN[7] proposed a deep matching network based on a multi-layer attention mechanism, indicating that multiple stacked attention layers can improve the performance of the model. OSOA-DFN[8] made further improvements on AF-DMN, making full use of the original semantics of the sample itself, and still achieving good results using a single-layer self-attention mechanism. The RE2[9] model proposed by Yang et al. realizes multi-layer rich feature matching, which is simple and efficient. Recently, the fine-tuning program Bert

[10] based on pre-trained language models has achieved state-of-the-art results in matching tasks. However, due to its large number of parameters and slow inference speed, it cannot be directly deployed in actual applications. Different from the above work, we propose a deep learning text matching method based on enhanced grammatical information. When encoding sentences, we use the dependency syntactic information of sentence pairs, and use Euclidean distance to enhance the interactive matching process, guiding the model to make better matches.

[0005] [References]

[0006] [1]Yin W,Schütze H,Xiang B,et al.Abcnn:Attention-based convolutional neural network for modeling sentence pairs[J].Transactions of the Association for Computational Linguistics,2016,4:259-272.

[0007] [2]Wang Z,Hamza W,Florian R.Bilateral Multi-Perspective Matching for Natural Language Sentences[J].

[0008] [3]Tan C,Wei F,Wang W,et al.Multiway Attention Networks for ModelingSentence Pairs[C] / / IJCAI.2018:4411-4417.

[0009] [4]Tay Y,Luu A T,Hui S C.Compare,Compress and Propagate:EnhancingNeural Architectures with Alignment Factorization for Natural LanguageInference[C] / / Proceedings of the 2018 Conference on Empirical MethodsinNatural Language Processing.2018:1565-1575.

[0010] [5]Tay Y,TuanψL A, S C.Co-Stack Residual Affinity Networks withMulti-level Attention Refinement for Matching Text Sequences[J].

[0011] [6]Gong Y,Luo H,Zhang J.Natural language inference over interactionspace[J].arXiv preprint arXiv:1709.04348,2017.

[0012] [7]Duan C,Cui L,Chen X,et al.Attention-Fused Deep Matching Networkfor Natural Language Inference[C] / / IJCAI.2018:4033-4040.

[0013] [8]Liu M, Zhang Y, Xu J, et al.Original semantics-oriented attention and deep fusion network for sentence matching[C] / / Proceedings of the2019Conference on Empirical Methods in Natural Language Processing and the9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP).2019:2652-2661.

[0014] [9] Yang R, Zhang J, Gao X, et al. Simple and Effective Text Matching with RicherAlignment Features[C] / / Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 4699-4709.

[0015]

[10] Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deepbidirectional transformers for language understanding [J]. arXiv preprintarXiv:1810.04805, 2018. Summary of the invention

[0016] In order to solve the above problems, the present invention provides a deep learning text matching method based on the fusion of grammatical information in an enhanced manner.

[0017] On the one hand, this method focuses more on the semantic modeling of sentence pairs in the existing deep learning matching models, focusing on the syntactic information of sentences and the natural relative position relationship information between words in sentences formed by syntax. On the other hand, it overcomes the technical problem that the inner product matching method can only measure the similarity between sentence pairs by whether the directions of vectors are similar, and cannot capture the interaction between the relative position information at the word level and the syntactic information in the sentence pairs; the present invention avoids the loss of important syntactic position information in the final pooling.

[0018] The present invention is implemented by the following technical solutions:

[0019] A deep learning text matching method based on enhanced grammatical information fusion, the text matching method is established on the encoding layer, matching fusion layer and pooling prediction layer; the encoding layer is used to perform dependency syntactic analysis on the sentence when encoding the sentence pair, construct a dependency syntactic tree, and use a deep neural network to extract the dependency syntactic information and semantic information of the sentence, and add and fuse the two;

[0020] The matching fusion layer is used to calculate the Euclidean distance of the encoding features between the sentence pairs when calculating the similarity, subtract the corresponding positions of the inner product interaction matrix and the Euclidean distance matrix, and use the Euclidean distance to enhance the inner product interaction process;

[0021] The pooling prediction layer is used for extracting pooling features of information by using a pooling module that combines a weighted pooling method with a maximum pooling method.

[0022] Furthermore, the encoding layer performs a grammatical information enhancement process on the encoded representation of the sentence pair:

[0023] Embedding pre-trained word vectors;

[0024] BiLSTM is used to extract the contextual semantic representation information of the sentence, as shown in the following formula:

[0025]

[0026]

[0027] The dependency syntactic tree is constructed for the sentence pairs, and the dependency syntactic information of the sentence is extracted using TreeLSTM, which is then integrated with the semantic information, as shown in the following formula:

[0028]

[0029]

[0030]

[0031]

[0032] in: and They are p i (q j )The output of the hidden layer units of the left and right subtrees, after which the representation of the sentence will be transformed Hewei

[0033] Furthermore, the matching fusion layer performs a grammatical information enhancement process when calculating the similarity between texts:

[0034] Calculate the vector inner product interaction matrix and the Euclidean distance interaction matrix for the encoded information;

[0035] Subtract the corresponding positions of the two to obtain the enhanced interaction matrix, and then make a mutual representation between the sentence pairs;

[0036] Merge the representation information with the original encoding information;

[0037] The fused features are aggregated using BilSTM.

[0038] Furthermore, the pooling prediction layer enhances the grammatical information:

[0039] The semantic feature weights are normalized by the following formula to obtain the pooling feature;

[0040]

[0041]

[0042] The maximum pooling feature is obtained by splicing the pooling feature information through the following formula;

[0043]

[0044] Concatenate all pooled features to obtain the final pooled feature V;

[0045]

[0046] Beneficial Effects

[0047] The present invention can not only capture the semantic similarity between sentence pairs, but also better capture the fine-grained syntactic position information between sentence pair interactions, and enhance the interaction process by applying Euclidean distance. The present invention can more flexibly and stably integrate the semantic syntactic information between sentence pair contexts. Compared with the prior art, the present invention has verified the ability to accurately match sentence pairs on both SNLI and Quora datasets.

[0048] The present invention uses the pooling prediction layer to perform deep extraction of important features and key position information. Compared with the prior art, the present invention has a small number of parameters and excellent performance, and is feasible for deployment in practical applications, as shown in the following table:

[0049] Table 1: Examples

[0050] BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1It is a flow chart of the deep learning text matching method based on the enhanced method of fusing grammatical information of the present invention;

[0052] Figure 2 This is a model diagram of the deep learning text matching method based on the enhanced fusion of grammatical information in the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0054] like Figure 1 , Figure 2 As shown; the present invention protects a deep learning text matching method based on enhanced grammatical information fusion, which includes a text encoding layer based on dependency syntax enhancement, an information interaction layer based on Euclidean distance enhancement, an information fusion layer, and a pooling prediction layer based on feature weighted enhancement.

[0055] In the dependency syntax enhanced text encoding module, each word in the sentence is first represented by a pre-trained Glove word vector embedding with a dimension of 300. The Stanford neural network syntax parsing tool is used to parse the dependency syntax tree of the text, and then TreeLSTM is used to extract the syntactic features of the text, and BiLSTM is used to extract the semantic features of the text. The two are fused to obtain the overall features of the text.

[0056] The detailed steps are as follows:

[0057] 1. Input encoding layer

[0058] The input encoding layer mainly encodes sentence pairs. Given an input sentence pair sequence P = (p1, p2, p3, ..., p m ) and Q=(q1,q2,q3,...,q n ), p i and q j are the i-th and j-th words of P and Q. The present invention uses pre-trained word vectors as word embeddings and trains with the model. After embedding, the sentence pairs are converted into word encodings. In order to extract the semantic information of the sentence, the present invention uses a BiLSTM neural network to encode P and Q and concatenates the outputs of the hidden layer units as the semantic encoding representation of the sentence:

[0059]

[0060]

[0061] Afterwards, the Stanford neural network syntactic dependency parser is used to perform syntactic parsing on the sentence, and the syntactic position information is extracted from the dependency syntax tree of the sentence pair through the TreeLSTM neural network, and it is fused with the semantic coding to obtain the final coding output.

[0062]

[0063]

[0064]

[0065]

[0066] in and They are p i (q j )The output of the hidden layer units of the left and right subtrees, after which the representation of the sentence will be transformed Hewei The present invention uses both as inputs of the matching fusion layer.

[0067] 2. Matching Fusion Layer

[0068] The general baseline model performs interactive matching in the form of inner product:

[0069]

[0070] This inner product interactive matching method focuses on the directionality of the vector and can only judge whether the semantics of sentence pairs are similar based on the directionality of the vector, but cannot achieve the interaction of syntactic position information between sentence pairs. Therefore, the present invention proposes to calculate the Euclidean distance between sentence pairs to enhance the inner product interaction matrix.

[0071]

[0072] A=C pq +E pq #(9)

[0073] A∈R m×n It is the enhanced interaction matrix, and the obtained interaction matrix is ​​used to represent the mutual encoding between sentence pairs.

[0074]

[0075]

[0076] m and n are the lengths of the two sentences respectively, and It is the encoded representation after interaction.

[0077] The present invention applies a simple but effective fusion module to fuse the aligned features with the original representation. The output of the fusion layer is calculated as follows:

[0078]

[0079]

[0080]

[0081] in are learnable parameters. is the bias value, which is generally initialized to 0, [·;·;·;·;·;·] is the concatenation operation, ⊙ is the Hadamard product, and the same operation is applied to sentence Q. Through the aggregation of BiLSTM, the fusion features of P and Q are finally obtained. and Subtractive fusion is done to highlight the difference between the aligned features and the original features, and Hadamard product fusion is done to highlight the similarity between the aligned features and the original features. Therefore, the interaction matrix enhanced by Euclidean distance can further enhance the difference and similarity between the interacted features and the original features during the fusion process.

[0082] 3. Pooling prediction layer

[0083] In order to reduce the loss of syntactic position feature information in the pooling module, the present invention makes improvements based on the original general pooling operation:

[0084]

[0085] The present invention refers to this pooling method as a feature-weighted pooling method. p ∈R m×d , The present invention performs the same operation on sentence Q. This pooling method can minimize information loss. At the same time, the present invention uses maximum pooling to highlight important feature information and concatenates the pooled feature information of the two to obtain the final pooled feature.

[0086]

[0087] Perform the same operation on sentence Q and concatenate all the pooled features as the final pooled feature V.

[0088]

[0089] The present invention puts V into the final multi-layer perceptron classifier. In the experiment, the multi-layer perceptron has a hidden layer with tanh activation and a softmax output layer. The entire model is trained end-to-end, and its objective function is a multi-classification cross entropy loss function.

[0090] Dataset Introduction:

[0091] Finally, the present invention evaluates our model on two large-scale baseline datasets: the SNLI dataset and the Quora dataset. SNLI is a benchmark dataset for natural language inference, which consists of 570,000 manually annotated sentence pairs, of which the training dataset has approximately 550,000 pairs, the evaluation dataset has 9,842 pairs, and the test set has 9,824 pairs. Each sentence pair is labeled with one of the following relationships, Y = {joint relationship, contradictory relationship, neutral relationship}, and we use accuracy as the evaluation indicator of this dataset. The Quora dataset is a dataset for identifying paraphrases, and each sample corresponds to a binary label 0 and 1, 1 represents that the sentence pairs can be paraphrased to each other, and 0 represents that the sentence pairs cannot be paraphrased to each other. The dataset consists of more than 400,000 real data, and accuracy is also used as an indicator of the dataset.

[0092] Implementation details:

[0093] First, the present invention uses the pytorch framework to implement the model and trains on the Nvidia 3090 GPU. The pre-trained word vector used is the 300-dimensional Glove 840B to initialize our word embedding and keep it updated during model training. The Stanford syntactic analyzer is used to perform dependency syntactic analysis on sentence pairs. We initialize all input and output dimensions and hidden layer data dimensions of BiLSMs and MLPs to 300 dimensions, and the hidden layer output of TreeLSTM is 600 dimensions. During training, we used the adam gradient optimization algorithm, the initial initial learning rate was 0.0004 and the exponential decay method, the first momentum was set to 0.9, the second momentum was set to 0.99, the batch size was 32, and the batch size for the Quora data set was 64. In order to avoid overfitting, on the SNLI data set, the present invention applied dropout 0.45 to all feedforward connections, and for the Quroa data set, the dropout was 0.3, and the present invention applied ReLU activation to all MLPs. The following table is the comparative experimental results of the model of the present invention and the baseline model.

[0094] Table 2 Performance of each model on the SNLI dataset

[0095]

[0096] Table 3 Performance of each model on the Quora dataset

[0097]

[0098]

[0099] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A deep learning text matching method based on enhanced grammatical information fusion, the text matching method is established in the encoding layer, matching fusion layer and pooling prediction layer; characterized in that: The encoding layer performs a grammatical information enhancement process on the encoded representation of the sentence pair: embedding the pre-trained word vector; BiLSTM is used to extract the contextual semantic representation information of the sentence, as shown in the following formula: The encoding layer is used to perform dependency syntactic analysis on the sentence when encoding the sentence pair, construct a dependency syntactic tree, use a deep neural network to extract the dependency syntactic information and semantic information of the sentence, and add and fuse the two to obtain the encoding feature; the encoding feature is: in: and They are p i (q j )The output of the hidden layer units of the left and right subtrees, after which the representation of the sentence will be transformed Hewei The matching fusion layer is used to calculate the Euclidean distance of the encoding features between the sentence pairs when calculating the similarity, subtract the corresponding positions of the inner product interaction matrix and the Euclidean distance matrix, and use the Euclidean distance to enhance the inner product interaction process to obtain the matching features; the matching features are: in: is a learnable parameter; is the bias value, initialized to 0, [·;·;·;·;·;·] is the concatenation operation, ⊙ is the Hadamard product, and the same operation is applied to sentence Q. Through the aggregation of BiLSTM, the fusion features of P and Q are finally obtained. and The pooling prediction layer uses a weighted pooling method and a maximum pooling method to combine the matching features to obtain the pooling feature output of the grammatical information, where: The pooling feature is obtained by normalizing the matching feature weights using the following formula; The maximum pooling feature is obtained by splicing the pooling feature information through the following formula; All the maximum pooling features are concatenated to obtain the pooling feature V output of grammatical information; 2. The deep learning text matching method based on enhanced grammatical information fusion according to claim 1 is characterized in that: The matching fusion layer performs a grammatical information enhancement process when calculating the similarity between texts: Calculate the vector inner product interaction matrix and the Euclidean distance interaction matrix for the encoded information; Subtract the corresponding positions of the two to obtain the enhanced interaction matrix, and then make a mutual representation between the sentence pairs; Merge the vector representation information with the original encoding information; The fused features are aggregated using BilSTM.