Dictionary translation method based on iterative dimension reduction

Abstract

The invention discloses a dictionary translation method based on iterative dimensionality reduction, which comprises the following steps of: collecting monolingual corpora of two languages, respectively carrying out word segmentation to obtain training data, and training word embedding spaces of the two languages; embedding the two monolingual words into dimensions which are respectively subjected to dimension reduction to a specified threshold value; embedding and inputting the words subjected to dimension reduction into an unsupervised dictionary translation algorithm for training, and storing two shared word embedding spaces; inferring an inter-translation dictionary of which the word frequency is greater than a specified threshold value in the two shared word embedding spaces by taking the CSLS as a measurement distance; inputting an inter-translation dictionary and dimensionality-reduced words into a supervised dictionary translation algorithm for training, and storing two shared word embedding spaces after model convergence; and repeating the steps until the model converges, and performing dictionary translation from the two shared word embedding to obtain a final bilingual dictionary. The method can effectively improve the dictionary translation accuracy of many remote languages, and can be widely applied to downstream tasks such as machine translation and knowledge mining.

Description

technical field [0001] The invention relates to a dictionary translation technology, in particular to a dictionary translation method based on iterative dimension reduction. Background technique [0002] Dictionary translation, also known as dictionary induction and dictionary translation, is a translation task that can be used to deal with word-level translation between different languages. In statistical machine translation, dictionary translation is a core task. It discovers mutually translated words from bilingual parallel corpora and is the main source of translation knowledge. In neural machine translation, dictionary translation is usually used in tasks such as unsupervised machine translation and multilingual machine translation. Here, words are represented by real vectors, which are word embeddings. All words are distributed in a multi-dimensional space, and the researchers found that the word embedding space shows a similar structure in some languages, which make...

AI Technical Summary

Problems solved by technology

[0004] In view of the poor stability of many current dictionary translation methods, good results have been achieved in similar language comparisons such as English-French and English-German, but the performance of long-distance language comparisons such as English-Chinese and English-Japanese still bad

Images