Bilingual corpus sentence alignment method and device, readable storage medium and computer equipment

Abstract

The invention relates to a bilingual corpus sentence alignment method and device, a computer readable storage medium and a computer device. The method comprises the steps of obtaining language types of a to-be-aligned parallel text and an original text and a translation text; preprocessing the to-be-aligned parallel text to obtain a to-be-aligned parallel sentence pair; calling a monolingual wordsegmentation model corresponding to the language types of the original text and the translation text from a monolingual word segmentation model group trained through a SentencePiece algorithm, and performing word segmentation processing to obtain a sentence segment group of the to-be-aligned parallel text and the sentence segment group of the translation text to be aligned; and performing format processing on the sentence fragment groups of the to-be-aligned original text and the to-be-aligned translated text according to a preset format processing mode to obtain bilingual sentence pair groups, calling a sentence alignment tool, and performing sentence alignment processing on the bilingual sentence pair groups according to the bilingual dictionary to obtain sentence alignment parallel corpora. The monolingual word segmentation models of all languages are trained through the SentencePiece algorithm, so that the coupling degree and maintenance difficulty of codes are reduced, and the maintenance cost is reduced.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a bilingual sentence alignment method, device, computer-readable storage medium and computer equipment. Background technique [0002] When performing sentence-level alignment in bilingual parallel corpora with chapter-level alignment, a feasible approach is to use sentence length information and lexical information to judge the similarity of sentences in the two language parallel corpora. [0003] For example, if the lengths of two sentences differ greatly, the similarity between the two sentences is low, and the possibility of being a parallel sentence pair is also small. Or, if the two sentences contain the same number or the same letter string, the similarity between the two sentences is higher, and the possibility that the two sentences are parallel sentence pairs is higher. And, when two sentences contain words of the same concept in two languages, the similarit...

AI Technical Summary

Problems solved by technology

[0005] Based on this, it is necessary to provide a bilingual sentence alignment method, device, computer-readable storage medium and computer equipment for the high maintenance cost of bilingual sentence alignment

Images