Method and device for domain-specific vocabulary discovery, and method and device for domain-specific vocabulary classifier training

Abstract

The invention discloses a method and a device for domain-specific vocabulary discovery, and a method and a device for domain-specific vocabulary classifier training. The method for domain-specific vocabulary discovery comprises: after a text is segmented to obtain a plurality of character string segments, selecting word character string segments from the character string segments; classifying vocabularies of a specific field and vocabularies of a general field from the word character string segments by using a domain-specific vocabulary classifier obtained by pre-training, wherein the classifier is trained by using a logistic regression model according to word forming feature vectors and specific field feature vectors of vocabularies in a general word list and a specific field word list. The methods and the devices can be applied to construct a complete, simple and efficient scheme for discovering vocabularies in a specific field and training a classifier.

Description

technical field [0001] The invention relates to the technical field of finding new words, in particular to a method and device for finding vocabulary in a specific field and training a classifier. Background technique [0002] With the development of information technology, electronic documents in various fields have become increasingly abundant. In various professional fields, the number of documents in the document database has shown an exponential growth trend, and the difficulty of processing these document information has become more and more difficult. Chinese has a special organizational structure, and its information processing has higher requirements for word segmentation technology, so Chinese documents are more difficult to process than English documents. How to find out the new professional vocabulary in a specific field of Chinese grammatical features plays an extremely important role in information retrieval. The specific field may be the financial field, the ...

AI Technical Summary

Problems solved by technology

[0004] The unsupervised word formation scheme based entirely on mutual information PMI, although the algorithm is simple and efficient, has two serious problems, one is the difficulty of threshold selection, and the other is that basic statistics cannot capture semantic information

[0005] The supervised new word discovery scheme mainly relies on two models of CRF and HMM. The effect of this scheme depends heavily on the training data, and the CRF and HMM models are relatively complicated.

Images