Korean word segmentation restoration method based on language model

Abstract

The invention discloses a Korean word segmentation restoration method based on a language model, and the method comprises the following steps: 1) language model training: carrying out the language model training of Korean monolingual data through employing a Unigram method, and obtaining a Korean language model for subsequent word segmentation restoration operation; 2) bilingual data word segmentation: training a translation system, and performing word segmentation on bilingual training data by using a word segmentation tool; 3) translation model training: performing word segmentation processing on data by using the language model generated in the step 1), and then inputting the data after word segmentation into a neural network model to start training the model until the model converges;4, translated text word segmentation reduction: using the translation system obtained through training in the step 3 for translating test statements, and combining the obtained Korean translated textsinto a standard Korean writing method. By means of the method, the problems that word meaning granularity in Korean data is large, and the data are sparse can be relieved, and the quality of translated texts of machine translation with target languages being Korean is effectively improved.

Description

technical field [0001] The invention relates to a word segmentation restoration technology in language processing, in particular to a Korean word segmentation restoration method based on a language model. Background technique [0002] "Word" is the smallest language unit that can be used independently in a language. Words are usually used as the basic unit in the training process of machine translation. Any language needs to perform word segmentation on the data before training machine translation. Complete sentences are divided into groups of consecutive words, which are then trained for machine translation. [0003] There are two common word segmentation methods: one is inflectional words similar to English, and the words in the sentence are separated by spaces. This kind of data can be segmented using punctuation-based word segmentation. You only need to combine the punctuation marks in the sentence with the Words can be separated; another word segmentation method is sui...

AI Technical Summary

Problems solved by technology

[0018] Aiming at the deficiencies that the existing word segmentation and restoration methods cannot restore the translated Korean, which has a relatively large granularity, the problem to be solved by the present invention is to provide a method for word segmentation and restoration based on a language model, which can be used for the translation of the Korean language. Re-reduction processing after word segmentation for languages ​​with large granularity

Images