Mixed corpus word segment method based on LSTM (Long Short Term Memory)-CNN (Convolutional Neural Network)

Abstract

The invention discloses a mixed corpus word segment method based on an LSTM (Long Short Term Memory)-CNN (Convolutional Neural Network). The method comprises the steps of converting training mixed corpus data into the mixed corpus data at a character level; counting the mixed corpus data characters to obtain a character set, and numbering each character to obtain a character serial number set; counting character labels to obtain a label set, numbering the labels to obtain a label serial number set; segmenting the corpus according to a sentence length, and grouping the obtained sentences according to the sentence length to obtain a data set; randomly selecting a sentence subgroup from the data set, extracting a plurality of sentences from the sentence subgroup, wherein the characters of each sentence form a datum w, and a corresponding label set is y; converting the datum w into a corresponding serial number and sending the label y to a model LSTM-CNN, and training a parameter of a deeplearning model; and converting to-be-predicted mixed corpus data into data matched with the deep learning model, sending the to-be-predicted mixed corpus data to the trained deep learning model to obtain a word segment result.

Description

technical field [0001] The invention belongs to the technical field of computer software, and relates to a mixed corpus word segmentation method based on LSTM-CNN. Background technique [0002] Mixed corpus, in this patent application, refers to training or prediction data that includes corpus data in at least two languages. [0003] Word segmentation (Word Segment) refers to marking the input continuous string into a continuous label sequence according to the semantic information. In this patent application, the sequence data of Asian characters (simplified Chinese, traditional Chinese, Korean and Japanese) is divided into individual words, and spaces are used as the division between words. Registered words, in this patent, refer to words that have already appeared in the corpus vocabulary. Unregistered words refer to words that do not appear in the corpus vocabulary. [0004] The word segmentation method of mixed corpus involves two aspects of professional knowledge: on...

AI Technical Summary

Problems solved by technology

[0009] Disadvantage 1: The detection granularity of multiple languages ​​is not easy to distinguish, and there is a loss of participle accuracy because a certain language is not detected

[0010] Disadvantage 2: The dictionary-based method is too dependent on the dictionary, and cannot identify unregistered words that have not appeared in the dictionary based on semantic information

[0011] Disadvantage 3: The current statistics-based methods are mainly HMM (Hidden Markov) model and CRF (Conditional Random Field) model, because of the degree of calculation, it only considers the correlation between the current word and the previous word , the rest are conditionally independent, which is inconsistent with the reality, so there is room for further improvement in the accuracy of word segmentation

Images