Text similarity measuring system based on multi-feature fusion

Abstract

The invention provides a text similarity measuring system based on multi-feature fusion and relates to the field of intelligent information processing. According to the system, the text similarity is measured by fusing multiple features based on word frequencies, word vectors and Wikipedia labels. The invention aims to solve the problem of semantic loss caused by non-considering of contexts in a conventional text similarity measuring system and the problem of low similarity result accuracy caused by larger text length difference. The text similarity measuring system is implemented by the following steps: carrying out preprocessing such as word segmentation and stop word removal on a training text; training corpora of the processed training text as a word vector model; measuring the similarity based on the word frequencies, the similarity based on the word vectors and the similarity based on the Wikipedia labels between input text pairs to be computed, and carrying out weighted summation to obtain a final text semantic similarity measuring result. According to the system, the measurement accuracy of the text similarities can be improved, so that the requirement on intelligent information processing is met.

Description

technical field [0001] The invention relates to the technical field of intelligent information processing in the field of information technology, in particular to a method and system for measuring text similarity based on semantics. Background technique [0002] Semantic similarity is a core technology in the field of intelligent information processing, which can be applied to query expansion, word sense disambiguation, question answering system and information retrieval, etc. Assessing semantic similarity is also an important task in numerous research fields, such as psychology, cognitive science, artificial intelligence, etc. [0003] Supervised methods and unsupervised methods are two mainstream methods of semantic similarity measurement. Supervised methods require prior knowledge, such as knowledge base systems or ontology resources, such as DBPedia, WordNet, HowNet, etc.; unsupervised methods mainly use statistical learning The method obtains context information and ru...

AI Technical Summary

Problems solved by technology

[0016] The present invention provides a text similarity measurement system based on multi-feature fusion, which combines multiple features based on word frequency, word vector and Wikipedia tags to The purpose of measuring similarity is to solve the problem of lack of semantics caused by the lack of consideration of text context in conventional text similarity measurement systems, and the problem of low accuracy of similarity results caused by large differences in text length

Images