Problem semantic matching method for optimizing BERT

Abstract

The invention discloses a semantic matching method based on Bert, and the method is based on a pre-training model Bert-wwm-ext of Harbin Institute of Word, the model is firstly used to carry out unsupervised training of full word masks under our big data background, so that the model is firstly adapted to our data characteristics, and after the model based on our data is stored, the model based on our data is subjected to unsupervised training of full word masks under our big data background. The following adjustments are made on the structure of the model, a Pooling layer is added to an output layer of Bert, when sentences are input, each Batch inputs a group of specific sentences, a part of the sentences are similar in semantics, the remaining sentences are different in semantics, and in this way, the model is made to be similar to human learning, and the sentences can be input into the Bert. Contrast learning between data is considered, so that the model converges more quickly, after model architecture transformation is completed, sentence semantic similarity training is conducted again under the background of large corpora based on the model, comparison calculation between synonymous sentences and non-synonymous sentences is added in the training process, then the model is subjected to back propagation, and therefore the sentence semantic similarity is obtained. And finally obtained sentence vector semantic representation is more practical.

Description

technical field [0001] The invention relates to a Bert-based semantic matching technology and belongs to the field of artificial intelligence. Background technique [0002] Judging from the current mainstream technologies for text matching problem solutions, they can be summarized into three categories: statistical learning, deep learning, and transfer learning. Statistical learning technology routes mostly obtain text features through manual / statistical methods, and then compare the similarity between text pairs. Typical methods include but are not limited to: [0003] (1) Evaluation of similarity based on string operations, such as edit distance; [0004] (2) count the number of terms, and directly use statistical indicators such as similarity coefficients to calculate the similarity between the two; [0005] (3) Obtain the vector of text information by encoding such as word frequency-inverse text frequency (TF-IDF), and then obtain text similarity through inner product ...

AI Technical Summary

Problems solved by technology

Although the BERT-based model has achieved good performance in many NLP tasks, the quality of the sentence vector derived by itself is low, and it is difficult to reflect the semantic similarity between two sentences

Images