A Chinese word segmentation method based on deep learning

Abstract

The invention discloses a Chinese word segmentation method based on deep learning, which includes the following steps: mapping Chinese characters into literal vectors based on literal word frequencies; refining the literal vectors, and extracting feature vectors carrying contextual semantic information and character-carrying properties The feature vector of the feature; the character-level vector is effectively fused into a word-level distributed representation, and then the fused candidate word vector is sent to the deep learning model to calculate the sentence score, decoded by the method of beam search, and finally obtained by the sentence score. Select the appropriate word segmentation result. In this way, the word segmentation task is freed from tedious feature engineering, better system performance can be obtained by extracting richer feature information, and the complete segmentation history is used for modeling, with sequence-level word segmentation capabilities.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a Chinese word segmentation method based on deep learning. Background technique [0002] In the current big data environment, with the rapid development of IoT data perception, data cloud computing, triple play, and mobile Internet, the amount of data, especially unstructured text, has grown exponentially, and the types of data are diverse and heterogeneous. characteristics such as fragmentation, information fragmentation and low value density. The rapid expansion of data has brought great challenges to the automatic processing of information. How to efficiently and accurately process massive texts and extract valuable information has become an important topic in Natural Language Processing (NLP). [0003] In the field of natural language processing, especially in Chinese natural language processing, word segmentation is an important benchmark task, and the p...

AI Technical Summary

Problems solved by technology

[0005] Early Chinese word segmentation tasks based on deep learning used a simple feedback neural network to label each word in the training sequence. This method only obtains context information within a fixed window, and cannot learn the relationship between data and previous data well.

[0006] Recursive neural network can automatically learn more complex features by accumulating historical memory, making full use of context, but in practice, it is found that the recurrent neural network has the problem of gradient explosion and gradient disappearance, which makes it face the problem of not being able to perform well. Dealing with the problem of long-distance historical memory

Images