Electric power professional lexicon construction method based on hybrid model and clustering algorithm

Abstract

The invention relates to the field of artificial intelligence, in particular to an electric power professional lexicon construction method based on a hybrid model and a clustering algorithm. The method includes preprocessing the power text and the parallel corpus; performing word segmentation through a word segmentation model, performing word combination on a Jieba word segmentation result through a mutual information and left and right entropy algorithm and a TextRank algorithm, wherein text keywords are extracted from the Jieba word segmentation result through a TF-IDF algorithm and a Word2Vec word clustering algorithm, and text word segmentation is directly carried out through an information entropy word segmentation algorithm; summarizing the results, and comparing to obtain feature corpus words; selecting power professional words from the feature corpus words as seed words; meanwhile, using the exported power text lexicon as candidate words to perform word segmentation on the power text, and then using a word2vec algorithm to convert the words into word vectors; obtaining similar words through clustering, and then obtaining a power professional word library through rule filtering. According to the invention, most professional words in the non-power field can be filtered by using one clustering model, and the professional words are relatively complete.

Description

technical field [0001] The invention relates to the field of artificial intelligence, in particular to a method for constructing a power professional thesaurus based on a hybrid model and a clustering algorithm. Background technique [0002] In the Chinese language, the ideographic ability of single characters is poor and the meaning is scattered, while the ideographic ability of words is stronger and can describe a thing more accurately. Therefore, in natural language processing, words (including single characters) are usually the most basic processing unit. For languages ​​of the Latin family such as English, since there are spaces between words as word margins, words can be extracted simply and accurately. In the Chinese language, except for punctuation marks, words are closely connected without obvious boundaries, so it is difficult to extract words. Chinese word segmentation methods are roughly divided into three types: dictionary-based segmentation, statistical model...

AI Technical Summary

Problems solved by technology

The currently commonly used word segmentation methods are based on the artificial thesaurus, which can manually collect some commonly used words into the thesaurus, but this cannot cope with the endless new words, especially new words on the Internet.

Images