Word-splitting method based on Bi-LSTM-CNN

Abstract

The invention discloses a word-splitting method based on a Bi-LSTM-CNN. The method comprises the steps of converting training corpus data into character-level corpus data; counting corpus data characters to obtain a character set and numbering each character to obtain a character number set; counting character labels to obtain a label set, and numbering the labels to obtain a label number set; dividing a corpus according to the sentence length, grouping obtained sentences according to the length of the sentences to obtain a data set including n groups of sentences; selecting a group of sentences from the data set without putting back the sentences, extracting multiple sentences from the group, constituting data w through each characters of each sentence, and integrating the corresponding labels as y; converting data w into corresponding numbers and labels y, sending the data into the model Bi-LSTM-CNN, and training parameters of a depth leaning model; converting to-be-predicted data into data matched with the depth learning model, and sending the data to a trained depth learning model to obtain a word-splitting result.

Description

technical field [0001] The invention belongs to the technical field of computer software, and relates to a word segmentation method based on Bi-LSTM-CNN. Background technique [0002] In natural language processing, Asian texts do not have natural space separators like Western texts. Many Western text processing methods cannot be directly used for processing Asian texts (Chinese, Korean, and Japanese), because Asian texts (Chinese , Korean and Japanese) must go through the process of word segmentation in order to maintain consistency with Western languages. Therefore, word segmentation is the basis of information processing in the processing of Asian text, and its application scenarios include: [0003] 1. Search engine: An important function of a search engine is to do full-text indexing of documents. Its content is to segment the text, and then form an inverted index with the word segmentation results of the document and the document. When users query, they also first que...

AI Technical Summary

Problems solved by technology

The knowledge-based artificial neural network model has a small number of network layers in the actual application due to the gradient disappearance problem during model training, and the final word segmentation result has no obvious advantage

[0007] The word segmentation method based on the dictionary relies heavily on the dictionary library, the efficiency is relatively low, and cannot identify unregistered words; among the present invention, registered words refer to words that have appeared in the corpus vocabulary, and unregistered words refer to words that do not appear in word in corpus vocabulary

[0008] Based on the word frequency statistical word segmentation method (such as N-Gram), it can only associate the semantics of the first N-1 words of the current word, and the recognition accuracy is not high enough. When N increases, the efficiency is very low

And the recognition rate for unlogged is low

Images