A Chinese word segmentation method based on bidirectional lstm, cnn and crf

Abstract

The invention discloses a Chinese word segmentation method based on bidirectional LSTM, CNN and CRF, which is an improvement and optimization of traditional Chinese word segmentation based on a deep learning algorithm. The specific steps of the method are as follows: preprocessing the initial corpus, extracting the character feature information of the corpus and the corresponding pinyin feature information of the character; using the convolutional neural network to obtain the pinyin feature information vector of the character; using the word2vec model to obtain the character feature information vector of the text; Splicing the pinyin feature vector and the character feature vector to get the context information vector, put it into the bidirectional LSTM neural network; use the linear chain conditional random field to decode the output of the bidirectional LSTM to get the word segmentation tag sequence; decode the word segmentation tag sequence to get Word segmentation results. The present invention uses a deep neural network to extract text character features and pinyin features and combines conditional random fields for decoding, which can effectively extract Chinese text features and achieve good results in Chinese word segmentation tasks.

Description

technical field [0001] The invention relates to a Chinese word segmentation method based on bidirectional LSTM, CNN and CRF, belonging to the field of natural language processing. Background technique [0002] Chinese word segmentation is a basic task of Natural Language Processing (NLP). Its purpose is to split the input sequence of Chinese characters into individual words. [0003] In the field of Chinese word segmentation, traditional technologies can be divided into two categories. One is based on dictionaries and rules. It traverses Chinese character strings in a certain way and matches them with entries in the dictionary. If a certain string is found in the dictionary, Then the match is successful. The other is a method based on statistics. Related methods include conditional random field (CRF), hidden Markov model (HMM), and maximum entropy model (Maximum Entropy). Among them, conditional random field has been widely used in the field of Chinese word segmentation in...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the deficiencies of the above-mentioned prior art, the present invention provides a Chinese word segmentation method based on bidirectional LSTM, CNN and CRF, which effectively solves the problems of slow speed and low accuracy of traditional Chinese word segmentation in practical applications

Images