Text retrieval matching method and system

Abstract

The invention belongs to the field of artificial intelligence, and relates to a text retrieval matching method and system.The method comprises the steps that firstly, Chinese natural language reasoning text corpora in different fields are collected to serve as training corpora of a sentence representation model; step 2, training a sentence representation model in combination with a contrast learning method, and testing and screening out an optimal sentence representation model by using a sentence semantic matching reference data set; 3, performing similarity calculation on sentences in the long and short texts to be matched by using the screened optimal sentence representation mode; and 4, according to a similarity calculation result, obtaining matching scores of the sentences by adopting a Sigmod function to judge whether the sentences are similar sentences or not, and completing text retrieval. According to the length of the text and the characteristics of various models, the most suitable model architecture is used for carrying out the task of text retrieval, the problems that similarity of themes between sentences is lacked to be considered in text matching, the model length is limited and the like are solved, and the matching effect is improved.

Description

technical field [0001] The invention belongs to the field of artificial intelligence, and in particular relates to a text retrieval matching method and system. Background technique [0002] Text matching is a core task in natural language processing. Many natural language processing tasks can be abstracted into text matching problems. Whether in dialogue systems, recommendation systems, or search engines, text matching is essential. [0003] In the retrieval model, the traditional text matching method is to directly calculate the relevance ranking based on keyword retrieval or BM25 and other algorithms, but the disadvantage of this method is that it needs to maintain a large number of thesaurus databases and matching rules. Later, latent semantic analysis technologies such as LSA and LDA gradually appeared, trying to use the latent semantics hidden in the document to match the text, and can achieve better results than direct keyword matching. With the rise of deep learning,...

AI Technical Summary

Problems solved by technology

Although such a method simply and effectively learns the semantic relationship between sentences, there are two problems: 1) lack of similarity of topics between sentences, 2) the model has a length limit

[0004] In short text matching, the current technology often only considers the semantic similarity between sentences and ignores the correlation between the topics, and when dealing with long text matching, the past method is to stitch all sentences together, not only Does not take into account the limit of the length of the encoder, but also misses the semantic interaction information between sentences

Images