high-precision Thai sentence segmentation method

Abstract

The invention discloses a high-precision Thai sentence segmentation method, which comprises the following steps of: marking sentence boundary information by sentences in a Thai corpus to obtain required Thai sentences, and combining the required Thai sentences to obtain more Thai sentences with artificial sentence boundary marks; coding each Thai simple sentence in a character mode, generating a binary output string according to current sentence boundary information, and dividing a Thai corpus into a training set and a verification set according to the proportion of 9: 1; adopting a bidirectional RNN for building a Thai sentence segmentation model structure, adopting batch processing operation in the model training process, training a model, wherein each batch comprises a plurality of encoded Thai sentences, and performing pad mark filling on the portions, with the ends empty, of the rest sentences; and starting training by using the Thai sentence segmentation model to obtain a final Thai sentence segmentation model. Accurate positioning of the Thai sentence boundary is achieved, and the method is easy to achieve, accurate in segmentation, high in practicability and high in training speed.

Description

technical field [0001] The invention relates to the field of machine translation, in particular to a high-precision Thai sentence segmentation method. Background technique [0002] At present, machine translation technology has begun to translate around sentences as the basic unit. At this time, a very important task will be involved in the automatic judgment of sentence boundaries. In most of the data corpus that has not passed the data filtering step, the source of sentence pairs in it is rich and diverse, that is, there are sentences obtained through manual translation, and at the same time there are inter-translated sentence pairs obtained from the Internet relying on web crawlers. Therefore, the quality of sentence pairs in the corpus is often uneven, and sentences in the form of paragraphs can be seen almost everywhere, which may cause too many super-long sentences in the corpus. However, if such a situation occurs in the corpus, the training of the model and even the...

AI Technical Summary

Problems solved by technology

Although there is indeed a space (space) between most Thai sentences, there are also certain disadvantages

Facts have proved that not all Thai sentences have space symbols at the boundaries, which results in the fact that relying on this method to segment Thai sentences cannot achieve a very high precision, and there will be certain errors

Images