A word vector representation method based on Chinese language element and pinyin joint statistics

Abstract

A word vector representation method based on Chinese language element and pinyin joint statistics comprises the following steps of 1, collecting internet text information to construct a corpus, and conducting text cleaning and word segmentation processing on the constructed corpus; (2) carrying out word segmentation processing on the Chinese corpus, converting the processed Chinese corpus into pinyin information which does not reserve tone information, and then respectively carrying out statistical weights TFc, IDFc, TFp and IDFp on word frequency statistics and inverse document probability ofthe morphemes and pinyin characteristics in the training set corpus and the whole document; (3) constructing a Chinese single morpheme representation vector based on a Chinese word representation model of contextual morpheme and pinyin joint statistics; And (4) training a three-layer neural network on the basis of the step (3) for predicting the central target word. According to the method, the adaptability of an offline dictionary and the corpus data scale can be met, large-scale unlabeled internet information text data can be directly learned, the consideration of a conventional word embedding model on Chinese language difference characteristics can be improved, and the representation and recognition accuracy of wrongly written words can be improved.

Description

Technical field: [0001] The invention belongs to the technical field of natural language processing and relates to a Chinese word vector representation model, in particular to a word vector representation method based on joint statistics of Chinese morphemes and pinyin. Background technique: [0002] At present, natural language processing technology has been applied to various aspects, and word representation technology in text is a basic research in the field of natural language processing. Chinese word representation technology is to express Chinese characters in the form of data vectors and apply them to the neural network language model. Data representation is used as a preparatory work. The quality of its expression seriously affects the performance of language model learning and training and scene applications. [0003] Usually, the completion of text data analysis for natural language processing requires the mining of massive text corpus information. With the rapid g...

AI Technical Summary

Problems solved by technology

Typos will not only affect word segmentation results, but also affect the performance of the Chinese word vector representation model

[0004]Currently existing representation methods such as regular expressions, vector spaces, and word vectors cannot meet the adaptability of offline dictionaries and corpus data, and it is difficult to directly learn large-scale infinite Annotated Internet information text data

At the same time, the conventional word embedding model has poor compatibility with the characteristics of Chinese language differences, and the accuracy of representation and recognition of typos and words is low.

Images