A Chinese word segmentation method based on depth learning

Abstract

The invention discloses a Chinese word segmentation method based on depth learning, comprising the following steps: Chinese characters are maped into character vector based on literal character frequency; the character vector is refined to extract the feature vector with context semantic information and the feature vector with character feature; the character-level vectors are effectively fused with the word-level distributed representation, and then the fused candidate vectors are sent into the depth learning model to calculate the sentence scores, which are decoded by the cluster search method, and finally the appropriate word segmentation results are selected by the sentence scores. In this way, the task of word segmentation can be freed from the tedious feature engineering, better system performance can be obtained by extracting more abundant feature information, and the whole segmentation history can be used for modeling, which has the ability of word segmentation at the sequencelevel.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a Chinese word segmentation method based on deep learning. Background technique [0002] In the current big data environment, with the rapid development of Internet of Things data perception, data cloud computing, triple play and mobile Internet, data, especially unstructured text data, is growing exponentially and presents types of diversification and heterogeneity. characteristics such as globalization, information fragmentation, and low value density. The rapid expansion of data has brought great challenges to the automatic processing of information. How to efficiently and accurately process massive texts and extract valuable information has become an important topic of Natural Language Processing (NLP). [0003] In the field of natural language processing, especially in Chinese natural language processing, word segmentation is an important benchmark task,...

AI Technical Summary

Problems solved by technology

[0005] Early Chinese word segmentation tasks based on deep learning used a simple feedback neural network to label each word in the training sequence. This method only obtains context information within a fixed window, and cannot learn the relationship between data and previous data well.

[0006] Recursive neural network can automatically learn more complex features by accumulating historical memory, making full use of context, but in practice, it is found that the recurrent neural network has the problem of gradient explosion and gradient disappearance, which makes it face the problem of not being able to perform well. Dealing with the problem of long-distance historical memory

Images