A multi-view interactive matching method and electronic device for noisy text

By employing a multi-view interactive matching method, utilizing scaling cosine attention mechanism and bidirectional attention interaction, the problem of disordered word order in noisy text matching is solved, improving matching accuracy and efficiency, enhancing the influence of key information, and outperforming traditional time-series models.

CN114626425BActive Publication Date: 2024-11-08NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011456860.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-10
Publication Date
2024-11-08
Estimated Expiration
2040-12-10

AI Technical Summary

Technical Problem

Existing methods for matching noisy text struggle to effectively identify and process text with disordered word order. Traditional time-series models are ineffective, and rule and feature engineering filtering methods have limited filtering capabilities.

Method used

A multi-view interactive matching method is adopted, which uses the scaling cosine attention mechanism to calculate the attention weights, suppresses noise and word order interference through bidirectional attention interaction and residual network, uses Word2vec and BERT models for preprocessing, and combines LSTM neural network for encoding to construct a noisy text interaction matrix and perform matching and scoring.

Benefits of technology

It improves the time efficiency and accuracy of noisy text matching, reduces the impact of disordered word order on matching results, enhances the weight of key information, avoids the problems of long document representation difficulties and long-distance dependencies, and improves matching performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114626425B_ABST
    Figure CN114626425B_ABST
Patent Text Reader

Abstract

The present invention provides a multi-view interactive matching method and electronic device for noise text, including encoding two segments of noise text to be matched respectively to obtain two segments of coding vector sequences, and adding position information to each coding vector of the two segments of coding vector sequences; performing internal interaction on the two segments of coding vector sequences with added position information to obtain two segments of internal interaction results respectively; performing external interaction on the two segments of internal interaction results to respectively construct two bidirectional noise text interaction matrices; splicing the two noise text interaction matrices to determine whether the two noise texts to be matched match. The present invention uses an attention mechanism to capture the bidirectional matching pattern between noise texts, is less affected by the logical order of sentences in the noise text, increases the influence of effective semantic words in the text, improves the time efficiency of the model and the noise text matching effect, and avoids the transfer matching problem.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a multi-view interactive matching method and an electronic device for noisy text. Background Art

[0002] There are a lot of noisy texts on the current Internet. Specifically, noisy texts refer to texts that contain meaningless text segments or have a disordered grammatical structure. There are two main problems in these noisy texts: in terms of content, the semantics expressed by the noisy texts are irrelevant to the original text, and are usually ambiguous and repetitive. In terms of form, the noisy texts have a relatively complex grammatical structure, and the sequential structure of the noisy texts is also diverse. In view of these two main problems in noisy texts, in order to solve the matching problem of noisy texts, it is necessary to design a matching model that is insensitive to noise and word order. The current mainstream noisy text matching method first uses rule methods and feature engineering to filter the noise in the text before matching. The filtered noisy text will be input into the time series matching model, which mainly includes Markov conditional random fields, recurrent neural networks, etc. The final model reads the input texts in sequence and obtains a matching score as a score for the matching result to determine whether the two noisy sentences match each other.

[0003] However, the filtering effect of using rules and feature engineering methods is limited. It is difficult to cover all noise instances and it is difficult to correctly identify all noise. Because the forms of noise vary, it is difficult to enumerate and summarize, and some noise still has practical significance in a specific context. In addition, because the sequential structure of the text largely determines the true meaning of the text, the disordered word order in the noisy text will lead to poor results of traditional time series models. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-view interactive matching method and electronic device for noisy text, which calculates the Attention weight by means of a scaled cosine attention mechanism and adopts an Attention weighted approach to suppress the interference of noise and word order on the text matching results, so that better text matching effects can still be obtained in the presence of text noise and word order disorder.

[0005] The technical solution of the present invention is:

[0006] A multi-view interactive matching method for noisy text, the steps of which include:

[0007] 1) Encode the two noise texts to be matched respectively to obtain two encoding vector sequences, and add position information to each encoding vector of the two encoding vector sequences;

[0008] 2) Performing internal interaction on the two coding vector sequences with position information added, and obtaining two internal interaction results respectively, wherein the dimensions of the internal interaction results and the coding vector sequence are consistent;

[0009] 3) By calculating the bidirectional attention distribution, the two internal interaction results are externally interacted and two bidirectional noisy text interaction matrices are constructed respectively;

[0010] 4) Concatenate the two noise text interaction matrices to determine whether the two noise texts to be matched match.

[0011] Furthermore, before encoding the two sections of noise text to be matched, the two sections of noise text to be matched are preprocessed; the preprocessing includes: removing punctuation marks, stop words and low-frequency words.

[0012] Furthermore, the method of encoding two noisy texts includes: encoding using a pre-trained Word2vec or Bert model.

[0013] Furthermore, position information is added to each encoding vector of the two encoding vector sequences through the production method of position vector encoding in the Bert model.

[0014] Furthermore, before the two encoding vector sequences with the position information added are internally interacted, the two encoding vector sequences with the position information added are mapped to a unified semantic space through the following steps:

[0015] 1) Input the two coding vector sequences with position information into the bidirectional LSTM neural network for secondary coding to obtain two final vector coding sequences;

[0016] 2) Each vector code in the two final vector code sequences is mapped through the same residual network, so that the two code vector sequences with position information added are mapped to a unified semantic space.

[0017] Furthermore, the vector dimension of the final vector encoding sequence is determined by the number of hidden layer units in the second LSTM encoding layer in the bidirectional LSTM neural network.

[0018] Furthermore, the noise internal interaction results are obtained through the following steps:

[0019] 1) Inputting the two encoding vector sequences with position information added into the first residual network respectively;

[0020] 2) Using the scaled cosine attention algorithm, the noise coding vector with position information added is used as the query item, and other vector coding sequences in the noise text are internally interacted to calculate the self-attention weight;

[0021] 3) Combining the encoding vector sequence with the position information added and the attention weight, obtaining a weighted encoding vector sequence;

[0022] 4) Sending the weighted encoding vector sequence to the second residual network to obtain an abstract vector representation of the noisy text;

[0023] 5) Perform L2 regularization on the abstract vector representation of the noisy text respectively to obtain the internal interaction result of the position information encoding vector sequence.

[0024] Furthermore, the noisy text interaction matrix is ​​constructed by the following steps:

[0025] 1) Take each encoding vector in the two internal interaction results as a query, perform external interaction based on cosine similarity attention on the internal interaction result of the other text, and obtain the attention weight distribution of the current encoding vector relative to the other encoding sequence;

[0026] 2) Use the attention weight distribution to weight the corresponding internal interaction results and obtain the external interaction vectors of each encoding vector in the internal interaction results;

[0027] 3) Inputting the vector sequence obtained according to the external interaction result into the third residual network, and performing L2 regularization operation on the output of the third residual network to obtain a noisy bidirectional interaction matrix.

[0028] Furthermore, the following steps are used to determine whether the two noise texts to be matched match:

[0029] 1) Obtain the concatenation result of two noise text interaction matrices;

[0030] 2) Input the splicing result into the scorer to obtain the matching score;

[0031] 3) Based on the matching score, determine whether the two noise texts to be matched match.

[0032] Furthermore, the structure of the scorer includes: a scoring network composed of fully connected layers.

[0033] A storage medium stores a computer program, wherein the computer program is configured to execute the method described above when running.

[0034] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer to execute the method described above.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] 1. This method mainly uses the attention mechanism to capture the matching patterns between noisy texts, and can improve the time efficiency of the model through parallel computing.

[0037] 2. The model is not a traditional time series model and is less affected by the logical order of sentences in the noisy text. This method is also effective for noisy text pairs with disordered word order.

[0038] 3. The scaled cosine attention mechanism can effectively suppress the interference of noise in the text in a weighted manner and increase the influence of valid semantic words in the text, thereby improving the matching effect of noisy text.

[0039] 4. In long-text matching, the attention mechanism can effectively avoid the problems of difficulty in representing long documents and long-distance dependencies in time series models, and the matching effect is significantly better than the matching method based on document representation.

[0040] 5. A two-way matching method is adopted, which not only calculates the matching pattern of noise text q to d, but also considers the matching degree of text q to d, so as to avoid the problem of transfer matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the multi-perspective interactive matching method for noisy text of the present invention.

[0042] Figure 2 This is a framework diagram of the multi-perspective interactive matching method for noisy text of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, principle, technical solution and advantages of the present invention more clearly understood, the present invention will be described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0044] The method provided by the present invention is applicable to the matching task between text pairs containing noisy texts and disordered word order. The main idea of ​​the method is to use multiple attention mechanisms to increase the weight of key information and reduce the interference of noise and sentence order, and capture the matching pattern of sentences in the interaction process, and finally the scoring network scores the matching results.

[0045] like Figure 1 , Figure 2As shown, the method of the present invention mainly includes three processes: internal interaction, external interaction and matching scoring. First, before these three processes, the text pair needs to be encoded. Given a noise query text q and its corresponding text to be matched d, the word vector sequence in the noise text pair will first be encoded by pre-training the existing Word2vec model to obtain a corresponding word vector sequence. After that, the word vector sequence will be encoded for the second time through the existing bidirectional LSTM neural network to obtain a vector encoding sequence whose dimension is determined by the number of LSTM hidden layer units. Each vector encoding in the final sequence will be mapped through the same residual network, so that the vector encoding sequences of the two texts are mapped to a unified semantic space.

[0046] The internal interaction module of the model mainly uses the scaled cosine similarity attention mechanism to capture the key information within the sentence and suppress the expression of secondary information by weighting the vector encoding sequence. After the internal interaction, the external interaction module of the model also uses the scaled cosine similarity mechanism to capture the interaction matching pattern between sentences based on the results of the internal interaction module. Finally, the scorer gives a matching score based on the results of the internal and external interactions.

[0047] The internal interaction process is defined as a Self-Attention process. The model adopts a scaled cosine attention mechanism, and the input is a vector code sequence of the text. For the noise text q, the internal interaction uses the words in the vector code sequence of the text q as the query items in the attention algorithm, and weights the vector codes of other words in the sequence. In this way, the Attention weight of the self-interaction of the text q can be obtained, and then the vector code of each word in the self-interaction result sequence is weighted according to the obtained Attention weight, so as to obtain the model's attention information on different word vectors to reduce the weight of noise words and increase the weight of key words. In the same way, the present invention performs Attention weighting on the vector code sequence in the candidate document d through the same scaled cosine attention mechanism. The calculation method of Attention can avoid the influence of the text order structure to a certain extent, because although the order of word positions in the same layer changes, the weights of the corresponding positions calculated by Attention are unchanged. The specific calculation process is as follows:

[0048]

[0049]

[0050]

[0051]

[0052] Where q and k are the query and key values ​​for calculating the scaled cosine Attention, W q and W k is a trainable parameter matrix, α ij is the correlation between the i-th word and the j-th word, I q is the word vector sequence of the stacked and weighted noise text. The above formula represents the self-interaction process of text q, where I q This is the final result of the self-interaction process, which is a sequence encoding of a word vector. In addition, the self-interaction result of candidate document d is also obtained in the same way.

[0053] The external interaction module of the model is defined as a bidirectional Attention interaction process, and the present invention also uses the scaled cosine Attention algorithm to implement it. The input of the bidirectional interaction layer is the output of the self-interaction layer, that is, the two-segment vector coding sequence model constructs a bidirectional interaction matrix by calculating the bidirectional attention distribution, so as to obtain the words that the current word vector focuses on in another noise text. Assuming that the length of the noise text q is n, and the length of its corresponding candidate document d is m, the model first uses the word vector of the text q to perform Attention on the word vector sequence in the candidate document d respectively, and performs weighted summation on the results of Attention, so that n fixed-dimensional interaction vectors can be obtained, and the dimension is equal to the dimension of the vector coding sequence of the candidate document d. In the same way, using the word vectors in the candidate document d to perform Attention on the word vector coding sequence of the text q and weighted summation, m interaction vectors with the same dimension as the vector coding sequence in the text q can be obtained. The detailed calculation process is as follows:

[0054] q=[q1;q2;...;q n ]=W q I q

[0055] k=[k1;k2;...;k n ]=W k I d

[0056]

[0057]

[0058] Among them I q , I d are the results obtained by self-interaction of text q and candidate document d, α ijis the correlation score between the i-th word in the noise text q and the j-th word in the text d, and O represents the interaction result from the noise text q to the candidate document d. Similarly, the interaction result from the noise document d to the candidate document q can also be obtained in the same way.

[0059] After obtaining the results of the two-way interaction, the results of the two-way interaction are spliced ​​and the final matching score is obtained through a fully connected scoring network, as follows:

[0060] f(p,e)=softmax(W y [u m ;u e ]+b y )

[0061] where u m ,u e They are the results of the noise text pairs q and d after external interaction, and then the dimension extracted by a single-layer convolutional neural network is fixed to d e The global feature representation vector of . e Represents the total number of convolution kernels. The width of the convolution kernel is consistent with the width of the external interaction result vector. The heights are 3, 5, and 7 respectively, and the number is the same. The feature vector obtained by each convolution kernel is subjected to the maximum pooling operation to obtain a constant term. W y and b y They are the weight matrix and bias term of the fully connected network respectively. The scoring function calculates the probability distribution of the labels of the matching noise text pairs (q, d), where the label of the text pair is 0 or 1 to indicate whether the two noise texts match.

[0062] During the training process, the present invention uses the hinge loss function to calculate the error and perform back propagation on the entire network. Given a noisy text pair, the objective function during training will minimize the hinge loss between the positive and negative examples, and define the projection interval between the positive and negative examples through an artificially set threshold. The loss function is defined as follows:

[0063]

[0064] Where P is the query set of noise text, E is the set of text to be matched, and e + ∈E represents the positive example document corresponding to the noise query text P, e - ∈E represents the corresponding negative example document, and γ represents the threshold hyperparameter set by the hinge loss function.

[0065] In one embodiment of the present invention, given a noise text q and a candidate document d, the specific process of calculating the matching score between the two texts is as follows:

[0066] (1) The noise texts q and d are segmented and punctuation marks, stop words and low-frequency words are removed. The lengths of the text sequences are n and m respectively.

[0067] (2) Encode each word in the noise text q and d through the Word2vec model, and convert each word in the text into a word with a length of d e A fixed vector.

[0068] (3) Generate corresponding position vectors for texts q and d by using the existing BERT model to encode position vectors, and convert the dimension d e The position vector of is concatenated with each word vector, and the position information is added to each word vector to obtain a word vector sequence containing the position information.

[0069] (4) This step is optional. The word vector after splicing position encoding is input into the bidirectional LSTM network, and the LSTM hidden layer state of each step is used as the encoding of the word vector. Adding this step can improve the accuracy of matching shorter texts.

[0070] (5) The word vectors (result of step 3) or LSTM output codes (result of step 4) encoding each splicing position in the noise texts q and d are input into the same residual network. The residual network can map words in different texts to the same vector representation space. Finally, the output of the residual network is a fixed-dimensional vector sequence with lengths n and m, respectively, where n and m are the lengths of the vector encoding sequences of the texts q and d, respectively.

[0071] (6) Use the scaled cosine attention algorithm to calculate the corresponding Self-Attention weight for the fixed-dimensional vector corresponding to each word in q and d.

[0072] (7) Use the Self-Attention weight calculated in the previous step to perform a dot product with the fixed-dimensional vector sequence after residual network mapping to obtain the weighted word vector.

[0073] (8) The weighted word vector obtained by self-interaction is input into a new residual network as the abstract representation of the self-interaction layer.

[0074] (9) Perform L2 regularization on the output of the residual network in the self-interaction layer to obtain the output of the self-interaction layer. The output of the self-interaction layer is still two vector sequences with lengths of n and m respectively, and the dimension of the output vector is consistent with the dimension of the word vector (d e +d m ) dimension, where d m Dimension to encode the position vector.

[0075] (10) Each output vector obtained from the self-interaction layer of the noise text q is used as the query to perform attention on the output vector sequence of the text d, and the attention weight of each word in the text q for the word vector sequence in the text d is obtained.

[0076] (11) Perform a dot product operation on the Attention weight calculated in the previous step and the word sequence after self-interaction in the candidate document d, weight the word sequence in d, and then accumulate the weighted results. Finally, the word vector in each noise text q can be calculated to obtain an interaction vector of fixed length.

[0077] (12) Each interaction vector is input into the same residual network, and the output of the residual network is subjected to L2 regularization, so that the external interaction result from the noise document q to the candidate document d is obtained as a vector sequence of length n.

[0078] (13) In a similar way, each word vector in the candidate document d is used as the query to pay attention to the word vector in the noise document q, and finally the external interaction result from the candidate document d to the noise document q is obtained as a vector sequence of length m.

[0079] (14) A convolutional neural network is used to encode the vector sequence obtained through external interaction, and finally two interaction vectors of fixed length are obtained.

[0080] (15) The two interaction vectors are concatenated and input into a fully connected neural network. The probability distribution of the category labels is calculated using the Softmax function. The label with the largest probability value is taken as the final matching result. The probability value corresponding to the result label is the confidence level of the matching result.

[0081] The above embodiments are only for better illustrating the purpose, principle, technical solution and beneficial effect of the present invention. It should be understood that the above embodiments are only specific embodiments of the present invention and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-view interactive matching method for noisy text, the steps of which include: 1) Encode the two noise texts to be matched respectively to obtain two encoding vector sequences, and add position information to each encoding vector of the two encoding vector sequences; 2) performing internal interaction on the two coding vector sequences with the position information added thereto, and obtaining two internal interaction results respectively, wherein the dimensions of the internal interaction result and the coding vector sequence are consistent, and performing internal interaction on the two coding vector sequences with the position information added thereto, and obtaining two internal interaction results respectively, include: Input the two encoding vector sequences with position information into the first residual network respectively; Through the scaled cosine attention algorithm, the noise encoding vector with position information is used as the query item, and other vector encoding sequences in the noise text are internally interacted to calculate the self-attention weight; Combine the encoding vector sequence with the position information and the attention weight to obtain a weighted encoding vector sequence; The weighted encoding vector sequence is sent to the second residual network to obtain an abstract vector representation of the noisy text; Perform L2 regularization on the abstract vector representation of the noisy text to obtain the internal interaction result of the position information encoding vector sequence; 3) By calculating the bidirectional attention distribution, performing external interaction on the two internal interaction results, and respectively constructing two bidirectional noise text interaction matrices; wherein, by calculating the bidirectional attention distribution, performing external interaction on the two internal interaction results, and respectively constructing two bidirectional noise text interaction matrices, the method includes: Take each encoding vector in the two internal interaction results as a query, perform external interaction based on cosine similarity attention on the internal interaction result of the other text, and obtain the attention weight distribution of the current encoding vector relative to the other encoding sequence; Use the attention weight distribution to weight the corresponding internal interaction results and obtain the external interaction vectors of each encoding vector in the internal interaction results; Input the vector sequence obtained according to the external interaction result into the third residual network, and perform L2 regularization operation on the output of the third residual network to obtain a noisy bidirectional interaction matrix; 4) Concatenate the two noise text interaction matrices to determine whether the two noise texts to be matched match.

2. The method according to claim 1, characterized in that Before encoding the two noise texts to be matched, the two noise texts to be matched are preprocessed; the preprocessing includes: removing punctuation marks, stop words and low-frequency words.

3. The method according to claim 1, characterized in that The method for encoding two sections of noisy text includes: encoding using a pre-trained Word2vec or Bert model; adding position information to each encoding vector of the two encoding vector sequences through the production method of position vector encoding in the Bert model.

4. The method according to claim 1, characterized in that Before the two encoding vector sequences with position information added are internally interacted, the two encoding vector sequences with position information added are mapped to a unified semantic space through the following steps: 1) Input the two coding vector sequences with position information into the bidirectional LSTM neural network for secondary coding to obtain two final vector coding sequences; 2) Each vector code in the two final vector code sequences is mapped through the same residual network, so that the two code vector sequences with position information added are mapped to a unified semantic space.

5. The method according to claim 4, characterized in that The vector dimension of the final vector encoding sequence is determined by the number of hidden layer units in the second LSTM encoding layer in the bidirectional LSTM neural network.

6. The method according to claim 1, characterized in that Determine whether the two noise texts to be matched match by following the steps below: 1) Obtain the concatenation result of two noise text interaction matrices; 2) Input the concatenation result into a scorer to obtain a matching score, wherein the structure of the scorer includes: a scoring network composed of a fully connected layer; 3) Based on the matching score, determine whether the two noise texts to be matched match.

7. A storage medium storing a computer program, wherein: The computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

8. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.