A matching method and system for short texts

Abstract

This invention discloses a matching method and system for short texts, including: acquiring a pair of texts from a corpus, each text comprising multiple sentences; copying each sentence to construct a training set; training the BERT model on the training set to obtain the final BERT model; training the similarity between the two sentences and training the cosine similarity of each sentence vector and its word embedding vector, as well as the weights of words in the sentence; inputting the word embedding vectors into the Transformer's encoding to obtain word position feature vectors; fusing the word position feature vectors and sentence vectors to obtain the final sentence vector; classifying the concatenated sentence vectors using a classification network to obtain the matching value of the input text pair; this invention improves the training model based on positive and negative examples, enabling similar sentences to obtain higher cosine similarity through the vectors output by the trained model, thus achieving more accurate and reasonable matching of the most similar texts.

Description

Technical Field

[0001] This invention relates to the field of text matching technology, and in particular to a matching method and system for short texts. Background Technology

[0002] Text matching based on sentence vector representation is an important fundamental problem in natural language processing (NLP) and can be applied to a large number of NLP tasks, such as information retrieval, question answering systems, paraphrasing, dialogue systems, and machine translation. These NLP tasks can be largely abstracted into text matching problems. For example, web search can be abstracted into a relevance matching problem between web pages and user search terms, automatic question answering can be abstracted into a satisfaction matching problem between candidate answers and questions, and text deduplication can be abstracted into a similarity matching problem between texts.

[0003] Sentence vector representation technology has always been a hot topic in the field of NLP. After obtaining sentence vectors through sentence vector representation technology, the similarity of sentences can be calculated or represented to a certain extent by calculating the cosine similarity between sentence vectors. A good sentence vector representation technology can match the most similar sentences in the corpus by calculating the similarity of sentences. In the pre-BERT era, word-embedding trained by word2vec combined with pooling strategies was generally used for sentence vector representation, or, when training data was available, TextCNN / BiLSTM combined with Siamese network strategies was used for sentence vector representation. In the BERT era, people took advantage of the inherent advantages of pre-trained language models and generally used the [CLS] vectors of the BERT model as sentence vector representations.

[0004] Traditional matching algorithms based on word overlap cannot effectively solve practical problems because they have significant limitations, including: 1. Semantic limitations: "taxi" and "cab" may not sound similar, but they refer to the same mode of transportation; "apple" can mean different things in different contexts, such as fruit or company; 2. Structural limitations: "machine learning" and "learning machine" may have completely overlapping words, but they convey different meanings; 3. Knowledge limitations: While lexical and syntactic terms may be correct, they are not appropriate when considered in the context of relevant knowledge. This indicates that text matching tasks cannot rely solely on literal matching; semantic matching is also essential.

[0005] For sentence vector representation methods under pre-trained models, the [CLS] vectors obtained by BERT itself already possess a certain degree of semantic information, which is based on BERT's built-in multi-head attention mechanism. However, for text similarity matching tasks, token-level matching cannot always accurately represent the similarity between texts, or in other words, [CLS] itself only contains a certain degree of semantic information. Therefore, additional training tasks are needed to strengthen the semantic representation of [CLS].

[0006] In general, the problems faced by existing technologies can be divided into two main aspects: First, when adding additional training tasks to the sentence vector representation of [CLS], a large amount of data may be required, and the data is generally sentences with certain relationships to facilitate matching in subsequent tasks, which greatly increases the cost of data. Second, [CLS] itself only contains partial information of the sentence, and may not be able to fully represent the semantic information of the entire sentence in some cases. Summary of the Invention

[0007] To address the shortcomings mentioned above, this invention provides a matching method and system for short texts.

[0008] To achieve the above objectives, the present invention provides a matching method for short texts, comprising:

[0009] Obtain a pair of texts from the corpus, the texts comprising multiple sentences, and copy each sentence once to construct a training set;

[0010] The training set is used to train the BERT model to obtain the final BERT model. The training includes training the similarity between two sentences and training the dot product or cosine similarity of each sentence vector and the embedding vector of each word in the sentence, as well as training the weight of the word in the sentence.

[0011] The word embedding vector is input into the Transformer's Encoding to obtain the word position feature vector. The word position feature vector and the sentence vector are then fused to obtain the final sentence vector.

[0012] The final sentence vector obtained by the connection is classified using a classification network to obtain the matching value of the input text pair.

[0013] Preferably, the similarity training between the two sentences includes:

[0014] Two outputs obtained from different dropout layers of the BERT model when the same sentence is input are considered positive examples.

[0015] Two outputs obtained from different dropout layers of the BERT model when the sentences are inputted are considered counterexamples.

[0016] The similarity between the positive examples and the negative examples is calculated separately, and the BERT model parameters are adjusted with the goal of increasing the similarity between the positive examples and decreasing the similarity between the negative examples.

[0017] Preferably, the formula for adjusting the BERT model parameters with the goal of increasing the similarity of positive examples and decreasing the similarity of negative examples is as follows:

[0018]

[0019] In the formula: the numerator of the logarithmic function is the positive example similarity; the denominator is the negative example similarity.

[0020] Preferably, the training of the dot product or cosine similarity of each sentence vector and its word embedding vector in the sentence, as well as the weights of the words in the sentence, includes:

[0021] The sentence is input into the BERT model to obtain the sentence vector and the embedding vector of each word in the sentence, and the dot product or cosine similarity between the two is calculated.

[0022] Extract the weight of each word by keyword;

[0023] After converting the dot product or cosine similarity and the word weights into probability distributions using a softmax operation, their KL divergence is calculated.

[0024] The parameters of the BERT model are adjusted with the goal of reducing the KL divergence.

[0025] Preferably, the KL divergence formula is:

[0026]

[0027] In the formula: w key It is the weight vector of the word; w cls It is the weight vector of the sentence.

[0028] The present invention also includes a matching system for short texts, comprising:

[0029] The acquisition module is used to acquire a pair of texts from the corpus, wherein the texts include multiple sentences, and each sentence is copied once to construct a training set;

[0030] The training module is used to train the BERT model on the training set to obtain the final BERT model. The training includes training the similarity between two sentences and training the dot product or cosine similarity of each sentence vector and the embedding vector of each word in the sentence, as well as training the weight of the word in the sentence.

[0031] The aggregation module is used to input the word embedding vector into the Transformer's Encoding to obtain the word position feature vector, and to fuse the word position feature vector and the sentence vector to obtain the final sentence vector.

[0032] The matching module is used to classify the final sentence vector obtained by the connection using a classification network to obtain the matching value of the input text pair.

[0033] Preferably, the similarity training between the two sentences includes:

[0034] Two outputs obtained from different dropout layers of the BERT model when the same sentence is input are considered positive examples.

[0035] Two outputs obtained from different dropout layers of the BERT model when the sentences are inputted are considered counterexamples.

[0036] The similarity between the positive examples and the negative examples is calculated separately, and the BERT model parameters are adjusted with the goal of increasing the similarity between the positive examples and decreasing the similarity between the negative examples.

[0037] Preferably, the formula for adjusting the BERT model parameters with the goal of increasing the similarity of positive examples and decreasing the similarity of negative examples is as follows:

[0038]

[0039] In the formula: the numerator of the logarithmic function is the positive example similarity; the denominator is the negative example similarity.

[0040] Preferably, the training of the dot product or cosine similarity of each sentence vector and its word embedding vector in the sentence, as well as the weights of the words in the sentence, includes:

[0041] The sentence is input into the BERT model to obtain the sentence vector and the embedding vector of each word in the sentence, and the dot product or cosine similarity between the two is calculated.

[0042] Extract the weight of each word by keyword;

[0043] After converting the dot product or cosine similarity and the word weights into probability distributions using a softmax operation, their KL divergence is calculated.

[0044] The parameters of the BERT model are adjusted with the goal of reducing the KL divergence.

[0045] Preferably, the KL divergence formula is:

[0046]

[0047] In the formula: w key It is the weight vector of the word; w cls It is the weight vector of the sentence.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] This invention improves the vector space of the sentence vectors output by the training model based on positive and negative examples, so that similar sentences can obtain higher cosine similarity through the vectors output by the training model. This can match the most similar text more accurately and reasonably. Attached Figure Description

[0050] Figure 1 This is a flowchart of the matching method for short texts according to the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Reference Figure 1 This invention provides a matching method for short texts, comprising:

[0053] Obtain a pair of texts from the corpus, each containing multiple sentences. Copy each sentence and use it to build a training set. This training set does not need to be labeled to meet the training requirements.

[0054] The training set is used to train the BERT model to obtain the final BERT model. The training includes training the similarity between two sentences, training the dot product or cosine similarity of each sentence vector and the embedding vector of each word in the sentence, and training the weight of each word in the sentence.

[0055] Input the word embedding vector into the Transformer's Encoding to obtain the word position feature vector, and fuse the word position feature vector and the sentence vector to obtain the final sentence vector;

[0056] A classification network is used to classify the final sentence vector obtained from the connections, thus obtaining the matching value of the input text pairs.

[0057] In this embodiment, the similarity training between the two sentences is called contrastive learning training. Contrastive learning is a self-supervised learning method used to learn general features of a dataset by having the model learn which data points are similar or different without labels. The key to contrastive learning is constructing positive and negative examples, including:

[0058] The same sentence input to the BERT model will produce two positive outputs from different dropout layers.

[0059] Two outputs from different dropout layers of the BERT model when inputting different sentences serve as counterexamples.

[0060] The similarity between positive and negative examples was calculated separately, and the BERT model parameters were adjusted with the goal of increasing the similarity between positive examples and decreasing the similarity between negative examples.

[0061] The formula for adjusting the BERT model parameters with the goal of increasing the similarity of positive examples and decreasing the similarity of negative examples is as follows:

[0062]

[0063] In the formula: the numerator of the logarithmic function is the positive example similarity; the denominator is the negative example similarity.

[0064] Furthermore, the training of each sentence vector and its dot product or cosine similarity to the embedding vectors of each word in the sentence, as well as the weights of the words in the sentence, includes:

[0065] The BERT model takes a sentence as input, obtains the sentence vector and the embedding vector of each word in the sentence, and calculates the dot product or cosine similarity between the two.

[0066] Extract the weight of each word by keyword;

[0067] After converting the dot product or cosine similarity and word weights into probability distributions using a softmax operation, their KL divergence is calculated.

[0068] The parameters of the BERT model were adjusted with the goal of reducing KL divergence.

[0069] The KL divergence formula is:

[0070]

[0071] In the formula: w key It is the weight vector of the word; w cls It is the weight vector of the sentence.

[0072] By fine-tuning the parameters of BERT through the above training, a BERT model with stronger text representation capabilities is obtained, thereby enabling the sentence vectors output by BERT to perform better on text matching tasks.

[0073] The present invention also includes a matching system for short texts, comprising:

[0074] The acquisition module is used to acquire a pair of texts from the corpus. The texts consist of multiple sentences. Each sentence is copied and used to build the training set.

[0075] The training module is used to train the BERT model on the training set to obtain the final BERT model. The training includes training the similarity between two sentences, training the dot product or cosine similarity of each sentence vector and the embedding vector of each word in the sentence, and training the weight of each word in the sentence.

[0076] The aggregation module is used to input the word embedding vector into the Transformer's Encoding to obtain the word position feature vector, and then fuse the word position feature vector and the sentence vector to obtain the final sentence vector.

[0077] The matching module is used to classify the sentence vectors obtained from the connections using a classification network to obtain the matching value of the input text pairs.

[0078] In this embodiment, the similarity training between the two sentences is called contrastive learning training. Contrastive learning is a self-supervised learning method used to learn general features of a dataset by having the model learn which data points are similar or different without labels. The key to contrastive learning is constructing positive and negative examples, including:

[0079] The same sentence input to the BERT model will produce two positive outputs from different dropout layers.

[0080] Two outputs from different dropout layers of the BERT model when inputting different sentences serve as counterexamples.

[0081] The similarity between positive and negative examples was calculated separately, and the BERT model parameters were adjusted with the goal of increasing the similarity between positive examples and decreasing the similarity between negative examples.

[0082] Furthermore, the training of each sentence vector and its dot product or cosine similarity to the embedding vectors of each word in the sentence, as well as the weights of the words in the sentence, includes:

[0083] The BERT model takes a sentence as input, obtains the sentence vector and the embedding vector of each word in the sentence, and calculates the dot product or cosine similarity between the two.

[0084] Extract the weight of each word by keyword;

[0085] After converting the dot product or cosine similarity and word weights into probability distributions using a softmax operation, their KL divergence is calculated.

[0086] The parameters of the BERT model were adjusted with the goal of reducing KL divergence.

[0087] By fine-tuning the parameters of BERT through the above training, a BERT model with stronger text representation capabilities is obtained, thereby enabling the sentence vectors output by BERT to perform better on text matching tasks.

[0088] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A matching method for short texts, characterized in that, include: Obtain a pair of texts from the corpus, the texts comprising multiple sentences, and copy each sentence once to construct a training set; The training set is used to train the BERT model to obtain the final BERT model, wherein the training includes: Using two identical / different sentences as input, the similarity training between the two sentences is performed with the goal of increasing / decreasing similarity. Training of the dot product or cosine similarity of each sentence vector and the word embedding vector in the sentence, as well as the weight of the word in the sentence: After converting the dot product or cosine similarity and the weight of the word into a probability distribution through a softmax operation, calculate their KL divergence; Adjust the parameters of the BERT model with the goal of reducing the KL divergence; The word embedding vector is input into the Transformer's Encoding to obtain the word position feature vector. The word position feature vector and the sentence vector are then fused to obtain the final sentence vector. The final sentence vector obtained by the connection is classified using a classification network to obtain the matching value of the input text pair.

2. The matching method for short texts according to claim 1, characterized in that, The similarity training between the two sentences includes: Two outputs obtained from different dropout layers of the BERT model when the same sentence is input are considered positive examples. Two outputs obtained from different dropout layers of the BERT model when the sentences are inputted are considered counterexamples. The similarity between the positive examples and the negative examples is calculated separately, and the BERT model parameters are adjusted with the goal of increasing the similarity between the positive examples and decreasing the similarity between the negative examples.

3. The matching method for short texts according to claim 2, characterized in that, The formula for adjusting the BERT model parameters with the goal of increasing the similarity of positive examples and decreasing the similarity of negative examples is as follows: ； In the formula: the numerator of the logarithmic function is the positive example similarity; the denominator is the negative example similarity.

4. The matching method for short texts according to claim 3, characterized in that, Training for each sentence vector and the dot product or cosine similarity of its word embedding vectors within the sentence, as well as the weights of the words in the sentence, includes: The sentence is input into the BERT model to obtain the sentence vector and the embedding vector of each word in the sentence, and the dot product or cosine similarity between the two is calculated. Extract the weight of each word by keyword.

5. The matching method for short texts according to claim 4, characterized in that, The KL divergence formula is: ； In the formula: w key It is the weight vector of the word; w cls It is the weight vector of the sentence.

6. A matching system for short texts, characterized in that, include: The acquisition module is used to acquire a pair of texts from the corpus, wherein the texts include multiple sentences, and each sentence is copied once to construct a training set; A training module is used to train the BERT model on the training set to obtain the final BERT model, wherein the training includes: Using two identical / different sentences as input, the similarity training between the two sentences is performed with the goal of increasing / decreasing similarity. Training of the dot product or cosine similarity of each sentence vector and the word embedding vector in the sentence, as well as the weight of the word in the sentence: After converting the dot product or cosine similarity and the weight of the word into a probability distribution through a softmax operation, calculate their KL divergence; Adjust the parameters of the BERT model with the goal of reducing the KL divergence; The aggregation module is used to input the word embedding vector into the Transformer's Encoding to obtain the word position feature vector, and to fuse the word position feature vector and the sentence vector to obtain the final sentence vector. The matching module is used to classify the final sentence vector obtained by the connection using a classification network to obtain the matching value of the input text pair.

7. The matching system for short texts according to claim 6, characterized in that, The similarity training between the two sentences includes: Two outputs obtained from different dropout layers of the BERT model when the same sentence is input are considered positive examples. Two outputs obtained from different dropout layers of the BERT model when the sentences are inputted are considered counterexamples. The similarity between the positive examples and the negative examples is calculated separately, and the BERT model parameters are adjusted with the goal of increasing the similarity between the positive examples and decreasing the similarity between the negative examples.

8. The matching system for short texts according to claim 7, characterized in that, The formula for adjusting the BERT model parameters with the goal of increasing the similarity of positive examples and decreasing the similarity of negative examples is as follows: ； In the formula: the numerator of the logarithmic function is the positive example similarity; the denominator is the negative example similarity.

9. The matching system for short texts according to claim 8, characterized in that, Training for each sentence vector and the dot product or cosine similarity of its word embedding vectors within the sentence, as well as the weights of the words in the sentence, includes: The sentence is input into the BERT model to obtain the sentence vector and the embedding vector of each word in the sentence, and the dot product or cosine similarity between the two is calculated. Extract the weight of each word by keyword; After converting the dot product or cosine similarity and the word weights into probability distributions using a softmax operation, their KL divergence is calculated. The parameters of the BERT model are adjusted with the goal of reducing the KL divergence.

10. The matching system for short texts according to claim 9, characterized in that, The KL divergence formula is: ； In the formula: w key It is the weight vector of the word; w cls It is the weight vector of the sentence.

Images