Chinese word segmentation method based on two-way LSTM, CNN and CRF

Abstract

The invention discloses a Chinese word segmentation method based on two-way LSTM, CNN and CRF which improves and optimizes traditional Chinese word segmentation base on deep learning algorithm. The method comprises following specific steps: preprocessing the initial corpus, extracting corpus character feature information and pinyin feature information corresponding to characters; using the convolutional neural network to obtain pinyin feature information vector of the characters; using the word2vec model to obtain the character feature information vector of text; splicing pinyin feature vectors and character feature vectors to obtain context information vectors and put the context information vectors to a bidirectional LSTM neural network; decoding the output of the bidirectional LSTM using the linear chain condition random field to obtain the word segmentation sequence; decoding the word segmentation label sequence to obtain word segmentation results. The invention utilizes the deep neural network to extract text character features and pinyin features and combines the conditional random field decoding, can effectively extract Chinese text features and achieve good effect on Chinese word segmentation tasks.

Description

technical field [0001] The invention relates to a Chinese word segmentation method based on bidirectional LSTM, CNN and CRF, and belongs to the field of natural language processing. Background technique [0002] Chinese word segmentation is a basic task of Natural Language Processing (NLP). Its purpose is to split the input Chinese character sequence into individual words. [0003] In the field of Chinese word segmentation, traditional technologies can be divided into two categories. One is the method based on dictionary and rules. It traverses the Chinese character string in a certain way to match the entry in the dictionary. If a certain string is found in the dictionary, The match is successful. The other method is based on statistics. Related methods include conditional random field (CRF), hidden Markov model (HMM), and maximum entropy model (Maximum Entropy). Among them, conditional random field has been widely used in the field of Chinese word segmentation in recent ...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the deficiencies of the above-mentioned prior art, the present invention provides a Chinese word segmentation method based on bidirectional LSTM, CNN and CRF, which effectively solves the problems of slow speed and low accuracy of traditional Chinese word segmentation in practical applications

Images