Open domain Chinese text naming entity identification method based on semi-supervised learning

Abstract

The invention discloses an open domain Chinese text naming entity identification method based on semi-supervised learning. The method comprises two steps of model training and prediction by a model. In the model training stage, a training set text is subjected to word segmentation preprocessing; then, in virtue of word vector space constructed by a word2vec tool, obtaining a word vector expressedby a word distribution type form in the training text; and utilizing the word vector in the training set and the existing entity type tag of each word vector to train a KNN (K-Nearest Neighbor) classifier and a CRF (Conditional Random Field) annotator, and generating a prediction model of a KNN-CRF naming entity type. In the model prediction stage, an empty reliable result set is imported, and when a new prediction result is generated by prediction, the prediction result is added into the reliable result set; when an amount in the reliable result set achieves a threshold value, previous KNN and CRF models are abandoned, the results in the reliable result set are added into the training set, and the KNN classifier and a CRF annotation model are trained again; and the above steps are repeated until a condition is met.

Description

Technical field: [0001] The invention is a named entity recognition method, especially an open domain named entity recognition method based on semi-supervised learning. Background technique: [0002] With the rapid development of information technology, today's society has ushered in a period of data explosion, and massive amounts of data are generated every moment, whether it is an individual, a business or a government. How to extract the valuable information contained in these data is very important. Named entity recognition is an application-driven discipline that uses computer technology to extract valuable information and knowledge from text data to solve this task. The traditional method of named entity recognition is keyword retrieval and related rules, such as matching with keywords in the database and fixed sentence pattern matching to extract target data. However, the disadvantage of this method is that it is difficult to ensure the integrity of the database dat...

AI Technical Summary

Problems solved by technology

However, the disadvantage of this method is that it is difficult to ensure the completeness of the database data and the rule-making process is time-consuming and labor-intensive, so further manual intervention is required to obtain the final required information

[0005] (2) There is a nesting phenomenon in the named entity

Images