Chinese word similarity calculation method based on fusion strategy

Abstract

The invention relates to a Chinese word similarity calculation method based on a fusion strategy. The method comprises steps of calculating word similarity based on the combination of four of HowNet,synonym forest, Word2Vec trained Chinese Wikipedia encyclopedia corpus and a Baidu dictionary; for two input words, firstly, judging whether the synonyms exist in a HowNet or synonym forest or not; ifyes, using the HowNet or synonym forest for calculating the similarity, if not, judging whether the HowNet or synonym forest exists in the Wikipedia corpus or the Baidu dictionary or not, and if yes,using the Word2vec or the Baidu dictionary for calculating the similarity of the words. The invention provides a Chinese word similarity calculation method based on a fusion strategy. According to the fusion strategy, the known network, the synonym forest, the word2vec and the Baidu dictionary are comprehensively considered, advantage complementation among strategies is formed, the calculated Spearman correlation coefficient and Pearson correlation coefficient are higher than those of other methods, the accuracy of a word similarity calculation result is improved, and the requirements of practical application can be well met.

Description

technical field [0001] The invention belongs to the technical field of text processing, and in particular relates to a method for calculating the similarity of Chinese words based on a fusion strategy. Background technique [0002] Word similarity calculation is a basic research topic of Chinese information processing. It has extensive and in-depth research in the fields of natural language processing, automatic question answering, knowledge graph, text classification, text clustering, information retrieval, information extraction, word sense disambiguation, and machine translation. Therefore, it has been studied and paid attention by more and more scholars. [0003] The current word similarity calculation can be divided into three types: methods based on existing knowledge ontology, methods based on large-scale corpus statistics, and word embedding methods based on corpus. The first method based on knowledge ontology uses the level, density and distance between words in th...

AI Technical Summary

Problems solved by technology

[0004] The disadvantage of the ontology-based method is that it is limited by the semantic dictionary and cannot handle unregistered (oov) words, and improper classification of words in the process of ontology construction will also lead to errors in the similarity calculation of words; based on large-scale corpus Statistical methods and word embedding methods are limited by the size of the corpus used for training, and the amount of calculation is relatively large, the calculation speed is slow, and the interference from the sparse corpus and the noise in the corpus is relatively large

Images