Semi-automatic word segmentation corpus labeling and training device

Abstract

The invention discloses a semi-automatic word segmentation corpus labeling and training device, which aims to overcome the defects of the corpora used during the word segmentation corpus labeling and training process. The device of the invention is realized through the following technical schemes of using a text corpus annotation preparation module for managing the to-be-annotated corpora and the segmented word corpora; based on a plurality of word segmentation algorithms, such as the bidirectional maximum matching word segmentation based on an integrated dictionary, CRF, JIEBA, etc., submitting the word segmentation annotation work of the raw corpus to a semi-automatic corpus word segmentation annotation module; creating the segmented word tagging tasks, selecting a labeling applicable algorithm model, carrying out the automatic annotations, on the basis of automatic labeling result fusion, feeding back a training model corpus and a labeling model generated by the text corpus labeling preparation module to the feedback model learning training module; selecting and carrying out model learning training, calling a unified training model interface to generate a core dictionary, updating a word segmentation training model table, establishing a labeling algorithm comprehensive evaluation model to evaluate a model labeling effect, so that a new word segmentation labeling task is completed.

Description

technical field [0001] The invention relates to the technical field of text mining, in particular to a semi-automatic labeling training device for word segmentation data. Background technique [0002] Words are the smallest, independently active, and meaningful language components, but there are no obvious distinguishing marks between words in Chinese. Therefore, Chinese word analysis is the basis and key of Chinese information processing. The accuracy of word segmentation is closely related to the accuracy of part-of-speech tagging. Organically integrating the process of word segmentation and part-of-speech tagging is conducive to eliminating ambiguity and improving overall efficiency. A Chinese sentence is composed of consecutive words, and there is no space separation between words. Part-of-speech tagging refers to the process of determining an appropriate part-of-speech for each word in a sentence. Chinese word segmentation is the first "process" of Chinese information...

AI Technical Summary

Problems solved by technology

Due to the generality and complexity of Chinese language knowledge, it is difficult to organize various language information into a form that can be directly read by machines. Therefore, the word segmentation system based on comprehension is still in the experimental stage.

Images