Sentence semantic distance measurement method

Abstract

The invention relates to a sentence semantic distance measurement method. The method comprises the following steps: firstly, carrying out word segmentation and stop word removal preprocessing on a sentence data set; selecting a word meaning similarity scheme, and setting a threshold value to execute normalization of synonymous words and synonymous words; then, calculating the vector space distanceof the two statements by combining smooth inverse frequency weighting and common component removal; measuring the word order distance of the two statements according to the out-of-order degree; calculating the semantic dependency distance of the two statements by combining the semantic dependency quintuple features; and finally, carrying out hybrid weighting calculation on the vector space distance, the word order distance and the semantic dependency distance. Measurement is from three dimensions of sentence vector representation, sentence word sequence and sentence component dependency, andfinally a final semantic distance is obtained in a weighted summation manner. A word level calculation means is utilized, and a sentence level operation idea is absorbed, and through introduction andcreative combination of a vector space distance, a word order distance and a semantic dependency distance, the semantic distance of the sentences is more comprehensively and reasonably measured.

Description

technical field [0001] The invention relates to a method for measuring the semantic distance of sentences, belonging to the technical field of text information processing. Background technique [0002] Semantic computing is one of the basic tasks in the field of text information processing, and it has practical usage scenarios at all levels from words, sentences, paragraphs to chapters. According to the development status of natural language processing technology, different ideas and strategies are currently adopted in semantic computing at different levels. For the calculation of semantic distance between sentences, the research work mainly focuses on two levels, namely word level and sentence level. [0003] The word-level measurement method, the main idea is to filter out the word strings that have a greater impact on semantics after the two sentences to be compared are segmented, and it is also possible to perform a shallow dependency analysis. For each word in the word...

AI Technical Summary

Problems solved by technology

[0006] The method of direct sentence embedding not only requires a very large-scale training corpus, but also uses a relatively complex neural network training model. Although it has achieved a small lead in some evaluation tasks, it is limited in the vertical field due to the limitations of corpus size and computing resources and Low interpretability, not universally applicable

Images