Chinese word segmentation method and system based on word vector representation learning

Abstract

The invention discloses a Chinese word segmentation method and system based on word vector representation learning. The method comprises the steps of performing preliminary word segmentation processing on a text to be subjected to word segmentation to obtain a preliminary word segmentation text; inputting the preliminary word segmentation text into a BERT model for training to obtain corpus word vectors; inputting the corpus word vectors into a Bi-GRU model for training to obtain a plurality of feature word vectors; calculating cosine similarity of two adjacent feature word vectors to obtain acosine similarity value; judging whether the cosine similarity value is greater than or equal to a preset threshold value or not; and if the cosine similarity value is greater than or equal to the preset threshold, combining the preliminary word segmentation results of the words corresponding to the two adjacent feature word vectors. By adopting the method and the system provided by the invention, the problem of word segmentation of the unregistered words in the specific field is solved, the word segmentation performance is improved, the semantic integrity of the words is not damaged by the word segmentation result, and a large amount of manpower tagging corpora can be avoided.

Description

technical field [0001] The invention relates to the technical field of word segmentation, in particular to a Chinese word segmentation method and system based on word vector representation learning. Background technique [0002] The accuracy rate of general-purpose word segmentation technology will drop significantly when it is applied in pre-specific field. In the general domain dictionary, that is, unregistered words, the specific domain corpus has a large number of unregistered words. For example, there are a large number of phrases such as "prefix + noun", "place name + noun" and "person name + noun" in the Tibetan animal husbandry corpus. Among them, the entity names of "prefix + noun" include Tibetan sheep, Tibetan pig, Tibetan snow chicken, Tibetan fennel, Tibetan fennel, saffron, etc.; the entity names of "place name + noun" include Zhongba Grassland, Plateau Rabbit, Sanjiang River Basin, and Alpine Vulture , Naqu Cordyceps, etc. The entity names of "person name + ...

AI Technical Summary

Problems solved by technology

Secondly, the word vector splits the semantics of the word itself, which destroys the integrity of the semantics of the word itself

In addition, word vector representation learning requires a lot of manpower to sequence the corpus, and domain word segmentation requires the construction of a domain dictionary, which is difficult and time-consuming

Images