Mixed corpus word segmentation method based on Bi-LSTM-CNN

Abstract

The invention discloses a mixed corpus word segmentation method based on a Bi-LSTM-CNN. According to the method, training mixed corpus data is converted into corpus data at a character level; statistical analysis is performed on characters of the corpus data to obtain a character set, and each character is numbered to obtain a character number set; statistical analysis is performed on character labels to obtain a label set, and the labels are numbered to obtain a label number set; a corpus is divided according to sentence length, obtained sentences are grouped according to the sentence length,and a data set is obtained; a sentence group is selected from the data set randomly without replacement, multiple sentences are extracted from the sentence group, the characters of each sentence forma piece of data w, and the corresponding label set is y; the data w is converted into corresponding numbers and labels y, the corresponding numbers and labels y are input into a model Bi-LSTM-CNN, and parameters of a deep learning model are trained; and to-be-predicted data is converted into data matched with the deep learning model, and the data is input into the trained deep learning model to obtain a word segmentation result.

Description

technical field [0001] The invention belongs to the technical field of computer software, and relates to a mixed corpus word segmentation method based on Bi-LSTM-CNN. Background technique [0002] Bi-LSTM, the full name in English is: Bi-directional Long Short-Term Memory, which means in Chinese, bidirectional long and short-term memory neural network. [0003] Mixed corpus, in this patent application, refers to training or prediction data that includes corpus data in at least two languages. [0004] Word segmentation (Word Segment) refers to marking the input continuous string into a continuous label sequence according to the semantic information. In this patent application, the sequence data of Asian characters (simplified Chinese, traditional Chinese, Korean and Japanese) is divided into individual words, and spaces are used as the division between words. Registered words, in this patent, refer to words that have already appeared in the corpus vocabulary. Unregistered ...

AI Technical Summary

Problems solved by technology

[0010] Disadvantage 1: The detection granularity of multiple languages ​​is not easy to distinguish, and there is a loss of participle accuracy because a certain language is not detected

[0011] Disadvantage 2: The dictionary-based method is too dependent on the dictionary, and cannot identify unregistered words that have not appeared in the dictionary based on semantic information

[0012] Disadvantage 3: The current statistics-based methods are mainly HMM (Hidden Markov) model and CRF (Conditional Random Field) model, because of the degree of calculation, it only considers the correlation between the current word and the previous word , the rest are conditionally independent, which is inconsistent with the reality, so there is room for further improvement in the accuracy of word segmentation

Images