Sentence similarity calculation method based on information amount

Abstract

The invention relates to a sentence similarity calculation method based on information amount. The method comprises the following steps: firstly, confirming the sense of a word according to the concept with the maximum information amount in words of two sentences, subsequently, calculating the information amount of the word and the public information amount of multiple words according to an hierarchical structure and corpus statistics of a semantic net, calculating the total information amount of multiple words by using the inclusion-exclusion principle in combinatorial mathematics so as to respectively obtain respective information amount of two sentences and the total information amount of two sentences, and finally defining and calculating the similarity of the sentences according to the Jaccard similarity principle. By adopting the method, the judgment of human beings on the similarity degree of sentences can be authentically simulated, moreover other natural language processing techniques such as corpus training parameters or experience parameters, dependence on scale of corpus, part-of-speech tagging and the like are not needed, the time performance is excellent, and quasi real-time calculation efficiency can be obtained on a conventional main current multi-core PC (Personal Computer) for sentence pairs of normal lengths.

Description

technical field [0001] The invention relates to a sentence similarity calculation method, in particular to an information-based sentence similarity calculation method, which belongs to the technical field of natural language processing. Background technique [0002] Sentence or short text similarity calculation is an important research content of natural language processing. In recent years, it has played an increasingly important role in information retrieval, machine translation, question answering system, automatic summarization and other application fields. The traditional method mostly uses the calculation method of document similarity, and only regards the sentence words as meaningless symbols that are not related to each other, which is not accurate enough for the calculation of sentences containing a small number of words. However, the commonly used hybrid methods usually need to train parameters on related data sets or use empirical parameters. The disadvantage is t...

AI Technical Summary

Problems solved by technology

Traditional methods mostly use the calculation method of document similarity, which only regards sentence words as meaningless symbols that are not related to each other, which is not accurate enough for calculating sentences containing a small number of words

However, the commonly used hybrid methods usually need to train parameters on related data sets or use empirical parameters. The disadvantage is that they rely on training data sets and are not very versatile.

Images