Chinese-Vietnamese parallel sentence pair extraction method based on cross-language bilingual pre-training and Bi-LSTM

Abstract

The invention relates to a Chinese-Vietnamese parallel sentence pair extraction method based on cross-language bilingual pre-training and Bi-LSTM and belongs to the technical field of natural languages. The method comprises the following steps: firstly, collecting Chinese-Vietnamese comparable corpora, and extracting Chinese-Vietnamese parallel sentence pairs from the comparable corpora; adding aChinese-Vietnamese bilingual dictionary and a large number of Chinese-Vietnamese monolanguages in pre-training, performing word alignment by mapping the Chinese-Vietnamese bilingual dictionaries to apublic semantic space, then generating a new dictionary iteratively in a self-learning mode by the Chinese-Vietnamese seed dictionary so that semantic similarity between Chinese-Vietnamese sentences is represented to the maximum extent; then inputting Chinese and Vietnamese sentences obtained after pre-training into a twin neural network composed of Bi-LSTM and CNN, and extracting global featuresand local features of the sentences; and finally, judging whether the input sentence pair is a Chinese-Vietnamese bilingual parallel sentence pair or not by using a full connection layer. Good effectis achieved in an experiment of extracting parallel sentence pairs from comparable corpora.

Description

technical field [0001] The invention relates to a Chinese-Vietnamese parallel sentence pair extraction method based on cross-language bilingual pre-training and Bi-LSTM, and belongs to the technical field of natural language processing. Background technique [0002] Parallel sentence pair extraction is an important method to alleviate the scarcity of machine translation data in natural language processing, aiming to expand Chinese-Vietnamese translation corpus. At present, parallel sentence pair extraction can be transformed into a sentence similarity classification task in the same semantic space, and its core lies in bilingual semantic space alignment. Traditional semantic space alignment methods rely on large-scale bilingual parallel corpora, but Vietnamese is relatively difficult to obtain large-scale parallel corpora as a low-resource scarce language, while it is relatively easy to obtain Chinese-Vietnamese monolinguals. Therefore, how to use Chinese-Vietnamese monolin...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a Chinese-Vietnamese parallel sentence pair extraction method based on cross-language bilingual pre-training and Bi-LSTM to solve the problem of scarcity of Chinese-Vietnamese bilingual parallel data, small-scale training data, and poor extraction effect of Chinese-Vietnamese bilingual parallel sentence pairs , so as to solve the problem of poor Chinese-Vietnamese machine translation

Images