Semi-automatic word segmentation material labeling training device

Abstract

The invention is a semi-automatic word segmentation material labeling training device, which aims to solve the drawbacks of using language material in the process of word segmentation material labeling and training. The present invention is realized through the following technical solutions: the text corpus labeling preparation module manages the labeling corpus and the word segmentation material, and through the two-way maximum matching word segmentation, CRF, JIEBA, etc. The work is submitted to the semi-automatic corpus word segmentation and tagging module to create a word segmentation and tagging task, select the appropriate algorithm model for labeling, and carry out automatic labeling. The feedback model learning and training module, select and model learning and training, call the unified training model interface to generate the core dictionary, update the word segmentation training model table, establish a comprehensive evaluation model of the labeling algorithm to evaluate the model labeling effect, and complete the new word segmentation labeling task.

Description

technical field [0001] The invention relates to the technical field of text mining, in particular to a semi-automatic labeling training device for word segmentation materials. Background technique [0002] Words are the smallest, independent and meaningful language components, but there is no obvious distinguishing mark between words in Chinese. Therefore, Chinese word analysis is the basis and key of Chinese information processing. The accuracy of word segmentation is closely related to the accuracy of part-of-speech tagging, and organically integrating the process of word segmentation and part-of-speech tagging is beneficial to eliminate ambiguity and improve overall efficiency. Chinese sentences are composed of consecutive words without spaces between them. Part-of-speech tagging refers to the process of determining an appropriate part of speech for each word in a sentence. Chinese word segmentation is the first "process" of Chinese information processing, and plays an ...

AI Technical Summary

Problems solved by technology

Due to the generality and complexity of Chinese language knowledge, it is difficult to organize various language information into a form that can be directly read by machines. Therefore, the word segmentation system based on comprehension is still in the experimental stage.

Images