Low resource domain word splitter training method based on transfer learning and word splitting method

Abstract

The invention discloses a low resource domain word splitter training method based on transfer learning and a word splitting method. The method comprises the steps of 1, conducting training in a targetdomain and all set domains respectively to generate corresponding word splitters; 2, using the word splitters of all the domains to conduct corpus word splitting processing on the target domain to obtain the implicit strata representation of each word xi on the corpus of the target domain of all the word splitters; 3, calculating the relevancy of the implicit strata representation of all the wordsplitters on the word xi and the implicit strata representation of the word splitters t of the target domain on the word xi, and obtaining weight vectors of the word splitters of all the domains on the word xi according to the relevancy; 4, weighing and summing the implicit strata representation of all the word splitters according to the weight vectors, obtaining the final implicit strata representation, and calculating labels of the word xi through the final implicit strata representation; 5, obtaining the word splitters of the target domain according to prediction labels and the standard result training of all words. By means of the low resource domain word splitter training method based on the transfer learning and the word splitting method, the word splitting effect of the low-resource domain corpus of the word splitters is greatly improved.

Description

technical field [0001] The invention belongs to the field of natural language processing, and relates to word segmentation of Chinese texts in a scene of insufficient resources, in particular to a training method and a word segmentation method for a word segmenter in a low-resource field based on transfer learning. Background technique [0002] For word segmentation tasks in the traditional news field, statistical methods have initially achieved good results, mainly including conditional random fields and perceptron models. However, these models need to extract a large number of features, so the generalization ability is limited. [0003] In recent years, more and more neural network-based methods have been used for automatic feature extraction, among which there are more word segmentation models, mainly including convolutional neural network (Convolutional Neural Network, CNN), long short-term memory neural network ( Long Short Term Memory Network, LSTM) etc. Although the...

AI Technical Summary

Problems solved by technology

Although these neural network-based methods are very effective, training these models and getting a better result requires a large amount of labeled data

For many specialized fields, the amount of labeled data is very limited, making it difficult to achieve better results by using neural network-based methods for word segmentation.

Images