Multi-criterion Chinese word segmentation method based on local self-attention mechanism and segmentation tree

Abstract

The invention discloses a multi-criterion Chinese word segmentation method based on a local attention mechanism and a segmentation tree. According to the method, for a text sequence of a corpus, the method comprises the following implementation steps: inputting a text sequence, obtaining unigram features and Bigram features of each character through word2vec, combining the unigram features and theBigram features with a predefined position vector to serve as an embedded layer, transmitting the embedded layer to a self-attention network, and obtaining the output of the embedded layer; and labelingeach character through crf layer decoding, and obtaininga plurality of labeling results; combining the labeling results into a segmentation tree to form a plurality of segmentation sequences; inputting the plurality of segmentation sequences into a scoring system, and selecting the group of segmentation sequences with the highest score as output. According to the method, the accuracy of multi-criterion word segmentation is improved.

Description

technical field [0001] The invention relates to a Chinese word segmentation task, specifically a multi-criteria Chinese word segmentation method based on a local self-attention mechanism and a segmentation tree, and belongs to the technical field of natural language processing. Background technique [0002] In recent years, Chinese word segmentation models based on neural networks have achieved very good results in word segmentation accuracy. At present, the best Chinese word segmentation method is based on a supervised machine learning algorithm, which relies heavily on large-scale and high-quality tagged corpora. However, the cost of building a high-quality annotated corpus is very large. Therefore, we need to combine the existing corpora, and although different corpora have different segmentation standards for some corpora, they still have many common features. Using this common feature can improve the accuracy of each word segmentation criterion. [0003] Existing solu...

AI Technical Summary

Problems solved by technology

[0003] Existing solutions can extract common features from multiple segmentation criteria, thereby improving the accuracy of each word segmentation criteria, but none of the current solutions can judge whether the segmented words are reasonable from the perspective of words, whether it is a real word or not. , and did not use the results of other word segmentation criteria

Images