A Method for Avoiding Duplication of Segments in Machine Translation

Abstract

The invention discloses a method for avoiding the repetition of machine translation translation segments, which belongs to the field of machine translation in natural language processing. In the decoding process of the greedy algorithm of machine translation, the duplicate segment detection mechanism of the translation is used to generate repeated target words. Probability is punished, and along with the length of repeated segment increases, carry out logarithmic level, linear level, exponential level penalty to target word generation probability successively, thereby reach the purpose of avoiding machine translation to generate repeated segment, stepwise punishment of the present invention, aggravate gradually, guarantee It does not affect the decoding process of the original translation too much, and can effectively reduce the false alarm rate of repeated segment penalties; the present invention fully considers the existence of repeated segments in the original text, and allows the length of repeated segments in the translation to be less than or equal to the length of the original text. Therefore, the consistency of the original text and the translation is guaranteed to the greatest extent, that is, the false alarm rate of the proposed method is reduced.

Description

technical field [0001] The invention relates to the field of machine translation in natural language processing, in particular to a method for avoiding repetition of machine translation translation segments. [0002] In the decoding process of machine translation, the repeat segment detection mechanism is used to penalize the generation probability of the repeated generated target words. As the length of repeated segments increases, punishment strategies such as logarithmic, linear, and exponential levels are used in turn to target words, so as to avoid the purpose of machine translation to generate repeated segments. Background technique [0003] With the further development of the trend of globalization and the proposal of the Belt and Road Initiative, machine translation has become an important research topic for communication between different language and ethnic groups. However, both academia and industry have found that machine translation based on deep learning metho...

AI Technical Summary

Problems solved by technology

[0012] The above-mentioned technology is mainly used in beam search (Beam Search) when evaluating multiple candidate translations. It has relatively large limitations and is not suitable for greedy algorithms and other fields. In addition, length penalty and repetition penalty are adopted. The two penalty factors function In the same function, it will lead to mutual influence, and the repeated punishment strategy cannot be fully effective

Images