Text similarity measurement method and device based on multi-model fusion

Abstract

The invention provides a text similarity measurement method based on multi-model fusion. The method comprises the following steps: preparing a training set and a test set; selecting four deep learning training models: Bert, Paddle, Xlnet and Tree-LSTM; for each training model, acquiring C sub-models; for each sub-model, calculating a similarity score and a loss function of the input data; evaluating the sub-models; selecting and fixing a group of super-parameter combination with the best evaluation value of each sub-model; continuously training each sub-model to converge a loss function, and storing the 4C sub-models at the moment; fusing the 4C sub-models by adopting a Boosting scheme so as to perform weighted addition on similarity scores of the sub-models to obtain a similarity measurement model; and testing and adjusting the similarity measurement model by using the data of the test set. According to the method, the accuracy of similarity measurement is effectively improved, the recall rate and accuracy of similarity judgment are improved, and the generalization ability of the model is improved.

Description

technical field [0001] The invention relates to a method and device for measuring text similarity based on multi-model fusion. Background technique [0002] Text similarity measurement refers to the measurement of the similarity between two texts, which has a wide range of applications in many fields. For example, in information retrieval, similarity can be used to identify similar words and improve the recall rate. In the automatic question answering scenario, the similarity can be used to calculate the matching degree between the user's question sentence in natural language and the question in the corpus, and the answer corresponding to the question with the highest matching degree is returned as the most responsive. However, in the application of machine translation, the bilingual translation is completed by analyzing the similarity of sentences. Whether the similarity can be accurately defined and calculated will affect the final translation effect. At this time, the us...

AI Technical Summary

Problems solved by technology

At the same time, some improvements have been made to the final similarity score, but this part of the improvement will bring a significant increase in time complexity

This scheme is relatively rough and has obvious shortcomings: First, it does not consider the semantics of documents, ignores the contextual relationship of words, and ignores the positional relationship of words, and only judges the similarity from the level of string comparison

[0006] As mentioned above, the existing technology is a series of solutions extended from the technical route based on keyword matching or vector space. The contextual relationship of words ignores the positional relationship of words. It should be noted that Chinese is extensive and profound. It is often impossible to infer the true meaning expressed in the document only by the literal meaning, which in turn affects the accuracy of similarity judgments.

Images