Multi-granularity word segmentation method and system based on sequence labeling modeling

Abstract

The invention relates to a multi-granularity word segmentation method and system based on sequence labeling modeling, and provides a method and system for acquiring a multi-granularity label sequenceby means of a machine learning method. The method comprises the steps that sentences in at least one single-granularity labeling data set are converted into word segmentation sequences complying withother n-1 word segmentation specifications respectively, n word segmentation sequences complying with the different specifications and corresponding to each sentence are converted into a multi-granularity word segmentation hierarchical structure, a multi-granularity label of each word in each sentence is obtained according to a predetermined coding method and the multi-granularity word segmentation hierarchical structures, and therefore a multi-granularity label sequence of each sentence is obtained; on the basis of the data set including the sentences and the corresponding multi-granularity label sequences, by training a sequence labeling model, a multi-granularity sequence labeling model is obtained. According to the multi-granularity word segmentation method and system based on sequencelabeling modeling, the concept of multi-granularity word segmentation is put forward for the first time, and the multi-granularity word segmentation hierarchical structures can be quickly and automatically obtained.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a multi-granularity word segmentation method and system based on sequence annotation modeling. Background technique [0002] Traditional word segmentation tasks are single-granularity word segmentation, that is, a continuous word sequence can only be recombined into a unique word sequence according to a specified specification. Multi-granularity word segmentation is to divide a continuous word sequence into multiple word sequences with different granularities according to different specifications. [0003] At present, the word segmentation tasks are all single-granularity word segmentation tasks, and at the same time, the existing manual tagged word segmentation data are all single-granularity word segmentation data. Therefore, there is no multi-granularity word segmentation data at home and abroad. The premise of multi-granularity word segmentation is a mul...

AI Technical Summary

Problems solved by technology

Since the existing word segmentation data is single-granularity word segmentation data, and there is no method for obtaining multi-granularity word segmentation data, one method is to complete the manual labeling method to obtain multi-granularity word segmentation data

However, the manual tagging method has the following disadvantages: (1) It is very difficult to formulate a multi-granularity word segmentation tagging specification, which is obviously more difficult than making a single-granularity word segmentation tagging specification

(2) The requirements for the annotator are higher, and the annotator needs to learn a more complex annotation specification

(3) The labeling process is more complicated, and the labeling result changes from a sequence structure to a hierarchical structure

In short, the human and time costs of manually labeling multi-granularity word segmentation data are very high

Images