Word segmentation method based on Bi-LSTM

Abstract

The invention discloses a word segmentation method based on Bi-LSTM. The method includes the steps that training corpus data is converted into character-level corpus data; the corpus data is segmentedaccording to sentence length, and several sentences are obtained; then the obtained sentences are grouped according to the sentence length, and a data set comprising n groups of sentences is obtained; several pieces of data are extracted from the data set to serve as iteration data; the each-time iteration data is converted into a fixed-length vector to be sent into a depth learning model Bi-LSTM, and parameters of the depth learning model Bi-LSTM are trained; when the loss value iteration change generated by the depth learning model is smaller than a set threshold value and is not decreasedany more or the maximum iteration time number is reached, training of the depth learning model is terminated, and the trained depth learning model Bi-LSTM is obtained; corpus data to be predicted is converted into character-level corpus data, the character-level corpus data is sent into the trained depth learning model Bi-LSTM, and a word segmentation result is obtained.

Description

technical field [0001] The invention belongs to the technical field of computer software, and relates to a word segmentation method based on Bi-LSTM. Background technique [0002] In natural language processing, Asian texts do not have natural space separators like Western texts. Many Western text processing methods cannot be directly used for processing Asian texts (Chinese, Korean, and Japanese), because Asian texts (Chinese , Korean and Japanese) must go through the process of word segmentation in order to maintain consistency with Western languages. Therefore, word segmentation is the basis of information processing in the processing of Asian text, and its application scenarios include: [0003] 1. Search engine: An important function of a search engine is to do full-text indexing of documents. Its content is to segment the text, and then form an inverted index with the word segmentation results of the document and the document. When users query, they also first query ...

AI Technical Summary

Problems solved by technology

The knowledge-based artificial neural network model has a small number of network layers in the actual application due to the gradient disappearance problem during model training, and the final word segmentation result has no obvious advantage

[0007] The word segmentation method based on the dictionary relies heavily on the dictionary library, the efficiency is relatively low, and cannot identify unregistered words; among the present invention, registered words refer to words that have appeared in the corpus vocabulary, and unregistered words refer to words that do not appear in word in corpus vocabulary

[0008] Based on the word frequency statistical word segmentation method (such as N-Gram), it can only associate the semantics of the first N-1 words of the current word, and the recognition accuracy is not high enough. When N increases, the efficiency is very low

And the recognition rate for unlogged is low

Images