Chinese word segmentation method

Abstract

Chinese word segmentation is a process that a Chinese character string is segmented into a word sequence according to a certain specification. Since the structure of a Chinese sentence is complex and no formal delimiters are in the presence between words and even the information of a next episode needs to be combined to carry out word segmentation judgment, the accuracy of an existing Chinese word segmentation method needs to be improved. The invention discloses a Chinese word segmentation method, which comprises the following steps that: 1: inputting a Chinese text to be subjected to word segmentation into a system to serve as a sequence A; 2: transferring the sequence A to a word vector searching layer, and converting an input character into a word vector to obtain a sequence B; 3: transferring the sequence B as an input sequence to a bidirectional shot and long term memory neuron network based on an attention mechanism, and subsequently, through one layer of hidden layer, obtaining an output sequence C; 4: transferring the sequence C as the input sequence to a conditional random field decoding layer, and generating a word segmentation markup tag sequence D; and finally, converting the sequence D into a text sequence E spaced by a space.

Description

technical field [0001] The invention belongs to the technical field of natural language processing and deep learning, and specifically relates to a Chinese word segmentation method based on an attention mechanism-based bidirectional long-short-term memory neural network and a conditional random field. Background technique [0002] Chinese word segmentation refers to the process of dividing continuous Chinese character strings into word sequences according to certain specifications. Chinese is different from English. Its own characteristic is that Chinese is based on words as the basic writing unit. Sentences and paragraphs are demarcated by delimiters, but there is no formal delimiter between words. In natural language processing, words It is the smallest meaningful language component that can act independently, so the quality of word segmentation directly affects the subsequent natural language processing tasks. Chinese word segmentation is an important basic research in...

AI Technical Summary

Problems solved by technology

However, the LSTM neural network can only remember the past information

Images