Text sequence annotating system and method based on Bi-LSTM (Bidirectional Long Short-Term Memory) and CRF (Conditional Random Field)

Abstract

The invention discloses a text sequence annotating system and method based on a Bi-LSTM (Bidirectional Long Short-Term Memory) and a CRF (Conditional Random Field). The system comprises a learning module and an annotating module, wherein the annotating module comprises a word segmenting module, a corpus annotating module and an adjusting and optimizing module; and the corpus annotating module comprises a part-of-speech annotating module and an entity recognizing module. The method comprises the following steps: preprocessing an obtained corpus; inputting the preprocessed corpus into a preset learning model; adjusting and saving parameters of the learning model; adding a corresponding prediction tag for the corpus respectively according to a sequence classifying result output by the learning model; performing word segmentation on an unknown corpus; initially annotating the unknown corpus being subjected to the word segmentation by using the adjusted learning model; adjusting and optimizing the initially-annotated unknow corpus; and finally annotating the adjusted and optimized corpus. Through adoption of the text sequence annotating system and method, a user can adjust a word library as required; a human-computer interactive adjusting function is realized; a process of automatically annotating in the same field and semi-automatically annotating in different fields is realized; the efficiency is increased; and the cost is lowered.

Description

technical field [0001] The invention relates to the field of information technology, in particular to a text sequence tagging system and method based on Bi-LSTM and CRF. Background technique [0002] With the development of the Internet, mobile Internet and big data technology, the scale of various text data resources has shown explosive growth, mainly including social media (such as Weibo account, official account, Facebook, Twitter, etc.) and news media (such as People's Daily , Phoenix News, Sohu News, etc.) websites, as well as semi-structured data on encyclopedia websites such as Baidu Encyclopedia and Wikipedia, Natural Language Processing (Natural Language Processing, NLP) plays a very important role in the process of text information extraction. important role. In the process of text mining, how to extract useful information from massive text data is of great value to enterprises or users. Sequence annotation is one of the most basic and commonly used NLP methods. ...

AI Technical Summary

Problems solved by technology

[0003] The current Chinese sequence labeling mainly has the following four problems in the application: first, the effect of Chinese word segmentation is not very good, such as the name "Wang Baoquan", if there is no name database or special processing (regularization or other grammatical processing), the name will be segmented into "wang" and "baoquan"

Second, most of the current Chinese sequence tagging methods adopt traditional methods such as Hidden Markov Model (HMM) or Conditional Random Field (CRF), although the effect obtained by adding a suitable lexicon is also It’s okay, but HMM’s ability to describe the sequence as a whole is weak, and CRF relies too much on the selection of features

Fourth, for different businesses, when it comes to different fields, it is necessary to manually label a corpus for model training. Since the model has particularly high requirements for training corpus, a large amount of accurately labeled corpus data is required, which will cost a lot of money. Human and material resources

Images