Field-specific vocabulary discovery and classifier training method and device

Abstract

The invention discloses a method and device for discovering vocabulary in a specific field and training a classifier. The method includes: after segmenting a text to obtain a number of character string fragments, selecting word string fragments; using the pre-trained A classifier of a specific domain vocabulary, which classifies a specific domain vocabulary and a general domain vocabulary from the word-forming string fragments; wherein, the classifier is based on the word-forming characteristics of the vocabulary in the general vocabulary and the specific domain vocabulary vectors and domain-specific feature vectors, trained using a logistic regression model. By applying the present invention, a complete, simple and highly efficient scheme for discovering vocabulary in a specific field and training a classifier can be constructed.

Description

technical field [0001] The invention relates to the technical field of finding new words, in particular to a method and device for finding vocabulary in a specific field and training a classifier. Background technique [0002] With the development of information technology, electronic documents in various fields have become increasingly abundant. In various professional fields, the number of documents in the document database has shown an exponential growth trend, and the difficulty of processing these document information has become more and more difficult. Chinese has a special organizational structure, and its information processing has higher requirements for word segmentation technology, so Chinese documents are more difficult to process than English documents. How to find out the new professional vocabulary in a specific field of Chinese grammatical features plays an extremely important role in information retrieval. The specific field may be the financial field, the ...

AI Technical Summary

Problems solved by technology

[0004] The unsupervised word formation scheme based entirely on mutual information PMI, although the algorithm is simple and efficient, has two serious problems, one is the difficulty of threshold selection, and the other is that basic statistics cannot capture semantic information

[0005] The supervised new word discovery scheme mainly relies on two models of CRF and HMM. The effect of this scheme depends heavily on the training data, and the CRF and HMM models are relatively complicated.

Images