LSTM-based mixed corpus word segmentation method

Abstract

The invention discloses an LSTM-based mixed corpus word segmentation method. According to the method, training mixed corpus data is converted into character-level mixed corpus data; and the mixed corpus data is divided according to sentence length to obtain a plurality of sentences; the obtained sentences are grouped according to the sentence length, and a data set comprising n groups of sentencesis obtained; a plurality of pieces of data are extracted from the data set to serve as iteration data; the data iterated each time is converted into vectors with fixed length, the vectors are input into a deep learning model LSTM, and parameters of the deep learning model LSTM are trained; when the iteration change of a loss value generated by the deep learning model is smaller than a set threshold, is no longer lowered or reaches the maximum number of iterations, training of the deep learning model is terminated, and a trained deep learning model LSTM is obtained; and to-be-predicted mixed corpus data is converted into character-level corpus data, the corpus data is input into the trained deep learning model LSTM, and a word segmentation result is obtained.

Description

technical field [0001] The invention belongs to the technical field of computer software, and relates to an LSTM-based mixed corpus word segmentation method. Background technique [0002] Mixed corpus, in this patent, refers to training or prediction data that includes corpus data in at least two languages. [0003] Word segmentation (Word Segment) refers to marking the input continuous string into a continuous label sequence according to the semantic information. In this patent, it refers to segmenting the sequence data of Asian type characters (simplified Chinese, traditional Chinese, Korean and Japanese) into individual words, and using spaces as the segmentation between words. [0004] The word segmentation method of mixed corpus involves two aspects of professional knowledge: on the one hand, the data format of multiple corpora is unified according to the character level; on the other hand, the professional knowledge involved is mainly sequence annotation in natural la...

AI Technical Summary

Problems solved by technology

[0014] Disadvantage 1: The detection granularity of multiple languages ​​is not easy to distinguish, and there is a loss of participle accuracy because a certain language is not detected

[0015] Disadvantage 2: The dictionary-based method is too dependent on the dictionary, and cannot identify unregistered words that have not appeared in the dictionary based on semantic information

[0016] Disadvantage 3: The current statistics-based methods are mainly HMM (Hidden Markov) model and CRF (Conditional Random Field) model, because of the degree of calculation responsibility, it only considers the correlation between the current word and the previous word , the rest are conditionally independent, which is inconsistent with the reality, so there is room for further improvement in the accuracy of word segmentation

Images