Trinity character annotation Chinese lexical analysis method based on Bi-LSTM-CRF

Abstract

The invention belongs to the technical field of lexical analysis, and discloses a trinity character annotation Chinese lexical analysis method based on Bi-LSTM-CRF, which comprises the steps of constructing a training corpus and a test corpus of a model based on an idea of trinity character annotation Chinese lexical analysis; building a Bi-LSTM-CRF model; inputting a training corpus, and trainingthe Bi-LSTM-CRF model through multiple iterations; segmenting an input Chinese text, and inputting the Chinese text into the trained model; determining a final lexical information marking sequence ofthe input Chinese text; and performing Chinese word segmentation, Chinese part-of-speech tagging and Chinese named entity recognition on the input text according to the lexical information tagging sequence to obtain a final Chinese lexical analysis result. According to the method, artificial feature engineering adopting traditional machine learning modeling is omitted, a representation learning method is introduced into machine learning, feature representation, extraction and selection can be automatically completed by the model, the efficiency is improved, and the Chinese lexical analysis precision is improved.

Description

technical field [0001] The invention belongs to the technical field of lexical analysis, and in particular relates to a Chinese lexical analysis method based on Bi-LSTM-CRF for trinity character labeling. Background technique [0002] In the field of Chinese information processing, Chinese lexical analysis is one of the important basic topics. It is not only the basis of deep Chinese information processing such as syntactic analysis, semantic analysis, and text understanding, but also a key link in applications such as machine translation, question answering systems, information extraction, and reading comprehension. Chinese lexical analysis mainly includes three sub-tasks of Chinese word segmentation, part-of-speech tagging and named entity recognition. There are two main concerns when analyzing the existing technical solutions below: (1) whether to process the three sub-tasks independently or three sub-tasks (2) Chinese lexical analysis modeling is based on traditional ma...

AI Technical Summary

Problems solved by technology

The literature [1] and patent literature [2] mentioned above are all implemented by traditional machine learning modeling. These methods require more manual feature engineering and consume more manpower.

Images