Word segmentation method based on common information and partial supervised learning of word segmentation tool

Abstract

The invention discloses a word segmentation method based on common information and partial supervised learning of a word segmentation tool. The method comprises the following steps: (1) pre-training a BiLSTM module with multiple word segmentation tool common information by using a large amount of unlabeled data and a BiLSTM neural network to obtain a trained BiLSTM neural network module; (2) training the initial word segmentation model by using a small amount of labeled data to obtain an initial word segmentation model M0 based on the convolutional neural network and the common information of various word segmentation tools; and (3) labeling a large amount of unlabeled data sets by using M0 to obtain a large amount of pseudo-label data, modifying a loss function in the M0, and jointly training the M0 after the loss function is modified by using a small amount of annotation data and a large amount of pseudo-tag data to obtain a Chinese word segmentation model M1 based on generality information and partial supervised learning of various word segmentation tools; (4) iterating the step (3) for n times to obtain a final word segmentation model Mn. According to the invention, the accuracy of cross-domain Chinese word segmentation is improved.

Description

Technical field [0001] The present invention relates to Chinese chord tasks, specifically, a word-based method based on word-of-term tools, and partially supervised learning methods, belonging to the technical field of natural language processing. Background technique [0002] In recent years, the Chinese-speaking model based on neural network has achieved a very good effect on the accuracy of words. However, the existing Chinese chronic method and word tools often decline sharply in special fields, and cross-domain is difficult to become Chinese word. In response to the lack of Chinese chronic issues in the field of lack of marked data, a combination of symbolic tool commonherent information and partial supervision learning. It combines a small number of target domain labeling data directly, and a training in a large number of target areas, and has gradually optimized models through iterative training, and improves the field of designation of word models. The method performs exp...

AI Technical Summary

Problems solved by technology

[0003] Aiming at the problems of insufficient tagging data and domain adaptability of existing Chinese word segmentation, the present invention discloses a cross-domain Chinese word segmentation method that integrates common information of multiple word segmentation tools and partly supervised learning, and improves the accuracy of cross-domain word segmentation

Images