A multilingual neural machine transliteration system and method

Abstract

The application discloses a multilingual neural machine transliteration system, which is characterized by being constructed based on a Transformer basic model architecture of a self-attention mechanism; comprising a plurality of language pair translation models constructed through a unified architecture; and being trained by using a data set containing a plurality of language pairs. The application also discloses a multilingual neural machine transliteration method using the above-mentioned multilingual neural machine transliteration system, which uses a writing system and / or a phonetic structure suitable for a corresponding language to perform word segmentation on a sentence, and adds a distinguishing symbol for identifying a current language at the beginning of a source language sentence and a target language sentence. The application applies a multilingual machine translation task to the transliteration field, and proposes an adaptive word segmentation method according to different languages. Experimental results on a public data set show that the multilingual neural machine transliteration is obviously superior to bilingual transliteration on medium and low resource languages.

Description

Technical Field

[0001] This invention relates to a machine translation system, and more particularly to a multilingual neural machine transliteration system and method. Background Technology

[0002] Machine translation (MT) is a method of translating one language into another using computer technology, employing human-defined rules, statistical information, or artificial neural networks. In the 21st century, the internet is undoubtedly the most widely used tool and the most prevalent communication platform globally. The greatest convenience brought by the internet lies in facilitating communication between different countries and regions, and this cross-regional and cross-linguistic communication has led to a surge in demand for machine translation.

[0003] The development of machine translation technology has gone through three stages. Initially, rule-based machine translation systems were used. Later, statistical information was used to complete machine translation tasks, a method known as statistical machine translation (SMT), which had significant advantages over rule-based machine translation. In recent years, as neural networks have gradually become the mainstream method in artificial intelligence, they have gradually replaced traditional statistical machine translation in machine translation. This method is now known as neural machine translation (NMT). Currently, neural machine translation has become the mainstream machine translation method.

[0004] Multilingual neural machine translation extends existing neural machine translation models to multiple languages ​​by modifying the dataset without altering the original model's structure.

[0005] While neural machine translation has demonstrated superior performance in text-level translation, it still has limitations when dealing with named entities such as personal names and place names. Current neural machine translation learns from the source language, mapping it to a suitable vector space to obtain its representation, and then reconstructs this representation to convert it into the target language. However, translating named entities typically requires a phonetic transliteration process based on their pronunciation in the source language, ensuring a similar pronunciation in the target language. This transliteration method demands that the neural machine translation model learn the phonemic information inherent in the language.

[0006] Most research on named entity transliteration focuses on using basic neural machine translation models, specifically variations of sequence-to-sequence model architectures, such as sequence-to-sequence models based on convolutional neural networks, recurrent neural networks, and long short-term memory networks. Some studies also employ training methods that combine the text to be translated with its corresponding syllable information, such as Pinyin, to utilize additional information and enhance the accuracy of named entity transliteration.

[0007] Existing methods for named entity transliteration mostly employ bilingual neural machine translation models, which puts pressure on the scalability of transliteration systems. If a transliteration system spanning multiple languages ​​is to be built, a neural machine translation model needs to be trained and maintained for each language pair, increasing the overall system overhead and making effective training and maintenance difficult. Furthermore, for some language pairs with scarce resources, there is insufficient corpus for training the neural machine translation model, leading to a significant performance degradation on these languages.

[0008] Current neural machine transliteration models often employ fixed segmentation methods, such as character-based segmentation, word-based segmentation, or sub-word segmentation. This type of segmentation often limits the processing capability of neural machine transliteration models for inputs from different languages. This is because the correspondence between phonemes and text often differs across languages, and applying the same segmentation method to all languages ​​for uniform processing often fails to achieve the desired results. For example, in Japanese katakana, each character represents a phoneme, while in English, multiple letters combine to form a phoneme. Summary of the Invention

[0009] This invention provides a multilingual neural machine translation system and method to solve the technical problems existing in the prior art.

[0010] The technical solution adopted by this invention to solve the technical problems existing in the prior art is: a multilingual neural machine translation system, which is built on the Transformer basic model architecture based on the self-attention mechanism; including multiple language pair translation models built through a unified architecture; and training the system using a dataset containing multiple language pairs.

[0011] Furthermore, the system includes an encoder assembly, which includes a multi-layer encoder. Each encoder layer includes a self-attention mechanism module A, a normalization module A, a feedforward neural network module A, and a normalization module B connected in sequence.

[0012] Furthermore, the system also includes a decoder component, which comprises a multi-layer decoder. Each layer of the decoder includes a self-attention mechanism module B, a normalization module C, a cross-attention mechanism module, a normalization module D, a feedforward neural network module B, and a normalization module E connected in sequence.

[0013] Furthermore, the translation models for multiple languages ​​were constructed using the fairseq framework.

[0014] The present invention also provides a multilingual neural machine translation method using the above-mentioned multilingual neural machine translation system, which uses a writing system and / or phonetic structure adapted to the corresponding language to segment the sentence into words, and adds a distinguishing symbol for the current language at the beginning of the source language sentence and the target language sentence, respectively.

[0015] Furthermore, when training the multilingual neural machine translation system, the source language sentences and target language sentences used as training samples are segmented according to the word segmentation methods suitable for their respective languages ​​before being used for training.

[0016] Furthermore, based on the size of the training samples, multiple language pairs in the sample dataset are divided into three groups: high-resource language group: training samples > 98K, including Russian, Arabic, Chinese and Japanese; medium-resource language group: training samples > 30K, including Thai, Hebrew, Korean and Persian; low-resource language group: training samples < 15K, other languages. A sampling strategy is used to balance the data distribution among different language pairs.

[0017] Furthermore, during training, the multilingual neural machine translation system uses the Adam optimizer to optimize parameters, setting β1 = 0.9, β2 = 0.98, and the label smoothing rate to 0.1. The learning rate is scaled based on the inverse square root of the number of running steps, with a warm-up step count of 4K and a weight decay rate of 0.0001. In the decoding phase, beam search decoding is performed with a beam size of 5 and a length penalty of 1.

[0018] The advantages and positive effects of this invention are: it applies multilingual machine translation tasks to the field of transliteration and proposes adaptive word segmentation methods based on different languages. Experimental results on public datasets show that multilingual neural machine transliteration significantly outperforms bilingual transliteration in both medium- and low-resource languages. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the present invention.

[0020] In the diagram: ×6 indicates the corresponding encoder and decoder layer number. Zh represents Chinese, and En represents English. <bos>Labels indicating the initial state of a sentence sequence. <eos>The label represents the termination state of the sentence sequence. Detailed Implementation

[0021] To further understand the invention's content, features, and effects, the following embodiments are provided, along with detailed descriptions in conjunction with the accompanying drawings:

[0022] The Chinese translations of some English words, phrases, and abbreviations used in this invention and its accompanying drawings are as follows:

[0023] Transformer: The Transformer model is a type of neural network that learns context and thus meaning by tracking relationships in sequential data.

[0024] Dropout: A module that is randomly deactivated.

[0025] Fairseq: An open-source framework from Facebook.

[0026] FFN: Feedforward Neural Network Module.

[0027] SentencePiece: Google's open-source subspeciesization module.

[0028] BPE: Byte-Pair Encoding Algorithm.

[0029] Unigram segmentation: a segmentation algorithm for unigram language models.

[0030] Adam: Adam optimizer.

[0031] NEWS2018: NEWS2018 dataset.

[0032] Unigram: A unigram language model.

[0033] Multilingual: A multilingual model.

[0034] Bilingual: a bilingual model.

[0035] CNN-LSTM: Convolutional Neural Network-Long Short-Term Memory Network Model (current multilingual neural machine transliteration model).

[0036] BiDeep+: A bidirectional deep long short-term memory network model (currently the best model).

[0037] Wiki-data: Data from Wikipedia.

[0038] System: List of models.

[0039] Arabic to English: A dataset for Arabic to English translation.

[0040] Subword: Sub-word.

[0041] Scheme: Segmentation mode.

[0042] Source: Source language.

[0043] Target: Target language.

[0044] Ours: The model proposed in this invention.

[0045] High: High-resource language.

[0046] Mid: Medium resource language.

[0047] Low: Low-resource language.

[0048] Avg: Average across all languages.

[0049] Please see Figure 1 A multilingual neural machine translation system is constructed based on a Transformer architecture with a self-attention mechanism; it includes multiple language pair translation models built with a unified architecture; and the system is trained using a dataset containing multiple language pairs.

[0050] Preferably, the system may include an encoder assembly, which includes a multi-layer encoder. Each encoder layer includes a self-attention mechanism module A, a normalization module A, a feedforward neural network module A, and a normalization module B connected in sequence. The encoder assembly may include 6-8 layers of encoders.

[0051] The source language is input into the first encoder of the encoder component through word embedding module A, and is output after being processed by multiple encoders.

[0052] Preferably, the system may further include a decoder component, which includes a multi-layer decoder. Each layer of the decoder includes a self-attention mechanism module B, a normalization module C, a cross-attention mechanism module, a normalization module D, a feedforward neural network module B, and a normalization module E connected in sequence. The decoder component may include 6-8 layers of decoders.

[0053] The first-layer decoder receives input from part of the output of the encoder component and part of the output of the last-layer decoder. The output of the last-layer decoder is then input to the first-layer decoder via the word embedding module B.

[0054] The decoders outside the first layer receive input from both the output of the encoder component and the output of the previous layer's decoder. After processing by the final decoder, the data passes sequentially through a fully connected layer and an excitation layer before being output as the target language output. The excitation layer can be processed using the softmax function.

[0055] Preferably, translation models for multiple languages ​​can be constructed using the fairseq framework.

[0056] The present invention also provides a multilingual neural machine translation method using the above-mentioned multilingual neural machine translation system, which uses a writing system and / or phonetic structure adapted to the corresponding language to segment the sentence into words, and adds a distinguishing symbol for the current language at the beginning of the source language sentence and the target language sentence, respectively.

[0057] Preferably, when training a multilingual neural machine translation system, the source language sentences and target language sentences used as training samples can be segmented according to the word segmentation methods suitable for their respective languages ​​before being used for training.

[0058] Preferably, multiple language pairs in the sample dataset can be divided into three groups according to the size of the training samples: high-resource language group: training samples > 98K, including Russian, Arabic, Chinese and Japanese; medium-resource language group: training samples > 30K, including Thai, Hebrew, Korean and Persian; low-resource language group: training samples < 15K, other languages ​​besides Russian, Arabic, Chinese, Japanese, Thai, Hebrew, Korean and Persian. A sampling strategy is used to balance the data distribution among different language pairs.

[0059] Preferably, during the training of the multilingual neural machine translation system, the Adam optimizer can be used to optimize the parameters, such that β1 = 0.9, β2 = 0.98, and the label smoothing rate is set to 0.1; the learning rate can be scaled according to the inverse square root of the number of running steps, the warm-up steps can be 4K, and the weight decay rate can be set to 0.0001; in the decoding stage, beam search decoding is performed, the beam size can be 5, and the length penalty can be 1.

[0060] The functional modules such as word embedding module A, self-attention mechanism module A, normalization module A, cross-attention mechanism module, normalization module B, feedforward neural network module A, normalization module C, linear layer, activation layer, word embedding module B, self-attention mechanism module B, normalization module D, feedforward neural network module B, and normalization module E can be applicable functional modules in the prior art, or can be constructed using existing hardware and software and conventional technical means.

[0061] The addition of letter codes after modules such as word embedding, normalization, feedforward neural network, and self-attention mechanism indicates their different positions within the system. Functional modules with different letter codes can use the same structure.

[0062] The workflow and working principle of the present invention will be further described below with reference to a preferred embodiment:

[0063] The working principle of this invention: Multilingual neural machine translation extends existing neural machine translation models to multiple languages ​​by modifying the dataset without altering the original model's structure. Specifically, assuming the current source language sentence X = {x 1 ,x 2 ,…,x n The corresponding target language sentence is Y = {y}. 1 ,y 2 ,…,y m The goal of neural machine translation is to translate the source language sentence X into the corresponding target language sentence Y. Multilingual machine translation, on the other hand, adds corresponding linguistic symbols, denoted as lang, to both the source language sentence and the target language sentence. src and lang tgt This transforms the original X and Y into:

[0064] X = {lang src ,x 1 ,x 2 ,…,x n };

[0065] Y = {lang tgt ,y 1 ,y 2 ,…,y m };

[0066] This method enables multilingual neural machine translation.

[0067] Similar to neural machine translation, multilingual machine translation employs an encoder-decoder architecture, in which language pairs are processed as follows:

[0068] h = Encoder([lang src ,x (i) ]);

[0069]

[0070] Where x (i) This represents the i-th source language sentence. This represents the first j words of the corresponding i-th target language sentence.

[0071] Next, the decoded information is mapped using the softmax function to obtain the corresponding probability values, thus yielding the overall probability of the target language sentence, as shown below:

[0072]

[0073]

[0074] This invention divides the 13 language pairs in the training dataset into three groups based on the size of the training data: high-resource languages ​​(>98K, including Russian, Arabic, Chinese, and Japanese), medium-resource languages ​​(>30K, including Thai, Hebrew, Korean, and Persian), and low-resource languages ​​(<15K, other languages). This invention employs a sampling strategy to balance the data distribution among different language pairs. For the multilingual transliteration model, a modified temperature-based sampling strategy is used before training. For the bilingual model, low-resource languages ​​are double-sampled. Due to the severe scarcity of Vietnamese (only 3K training instances), it is sampled 10 times. For BPE and unigram segmentation, the SentencePiece toolkit is used to preprocess multilingual named entities with a vocabulary of 32K.

[0075] This invention selects the Transformer architecture for multilingual and bilingual transliteration. Its encoder and decoder each have 6 layers, 8 attention heads, an embedding dimension of 512, and an FFN layer dimension of 2048. This invention sets the Dropout rate to 0.3 and applies it to the attention weights and residual connections. All models are implemented using the Fairseq framework.

[0076] This invention uses the Adam optimizer to optimize parameters (β1 = 0.9, β2 = 0.98), with a label smooth rate of 0.1. The learning rate is scaled based on the inverse square root of the number of training steps, with a warm-up of 4K steps and a weight decay rate of 0.0001. This invention sets the maximum number of labels in a batch to 4096 and the maximum number of training steps to 40K. During the decoding phase, this invention performs beam search decoding with a beam size of 5 and a length penalty of 1.

[0077] To avoid the negative impact of a uniform segmentation method on the performance of multilingual neural machine translation models, this invention proposes an adaptive segmentation approach. Since the languages ​​included in multilingual datasets have different writing systems and phonetic structures, applying a uniform segmentation strategy to all languages ​​may not be a good choice. Using a segmentation method that adapts to the writing system and phonetic structure of the respective language may be more preferable and flexible for neural translation. For example, character-based segmentation can be used for languages ​​like Chinese and Japanese, while sub-word-based segmentation can be used for languages ​​like English. This approach allows for more reasonable segmentation of different languages. Table 1 illustrates our proposed adaptive segmentation method.

[0078] Table 1: Examples of applying different word segmentation methods

[0079]

[0080] Table 1 shows the results after applying the adaptive word segmentation method. In the "Scheme" column, "S" represents the subword segmentation method, "C" represents the character segmentation method, and "C2S" means that the character segmentation method is used for the source language sentence, while the subword segmentation method is used for the target language sentence.

[0081] In the specific training and subsequent deployment, the text to be processed, i.e., the source language sentence and the target language sentence, is first segmented using the aforementioned adaptive word segmentation method, according to the segmentation methods suitable for both languages. The segmented text is then used as data for further processing. The specific processing procedure is similar to that of a multilingual neural machine translation model. Subsequently, the processed text is input into the system according to the current task, such as training or inference. Figure 1 The multilingual neural machine transliteration model shown yields the output results for the corresponding task.

[0082] This invention uses the publicly available NEWS2018 dataset to compare the performance of the proposed method with the best-performing models and current multilingual neural machine translation models. Furthermore, this invention uses the ANETAC dataset, ArabictoEnglish dataset, and Wiki-data dataset as supplementary datasets to compare the performance of the proposed model with bilingual models under the same configuration, as well as the performance of different adaptive word segmentation methods proposed in this invention.

[0083] Table 2 shows the transliteration accuracy on the NEWS2018 dataset.

[0084]

[0085] In Table 2:

[0086] En→XX indicates the direction of translation from English to other languages.

[0087] XX→En indicates the direction of translation from other languages ​​to English.

[0088] Zh represents Chinese characters.

[0089] Th represents Thai.

[0090] "He" is a Hebrew word.

[0091] Fa represents the Persian word.

[0092] Bn represents Bengali.

[0093] Hi is a Hindi word.

[0094] Ta represents the Tamil language.

[0095] Vi represents the Vietnamese language.

[0096] Kn represents the Kannada language.

[0097] In the table: BiDeep+ represents the current best model.

[0098] CNN-LSTM represents the current multilingual neural machine transliteration model.

[0099] Ours represents the model proposed in this invention.

[0100] The results of this invention on the NEWS2018 public dataset are shown in Table 2. The method proposed in this invention surpasses the existing multilingual neural machine translation model based on CNN-LSTM. Compared with the current best model, in the XX→En translation direction, the model of this invention surpasses the current best model in all language pairs except Chinese-English.

[0101] Table 3. Transliteration accuracy on the supplementary NEWS2018 dataset

[0102]

[0103] In Table 3:

[0104] En→XX indicates the direction of translation from English to other languages.

[0105] XX→En indicates the direction of translation from other languages ​​to English.

[0106] XX represents a collective term for other languages.

[0107] Ru represents the Russian language.

[0108] Ar represents Arabic.

[0109] Zh represents Chinese characters.

[0110] Ja represents Japanese.

[0111] "He" is a Hebrew word.

[0112] Th represents Thai.

[0113] Ko represents the Korean word.

[0114] Fa represents the Persian word.

[0115] Bn represents Bengali.

[0116] Hi is a Hindi word.

[0117] Ta represents the Tamil language.

[0118] Kn represents the Kannada language.

[0119] Vi represents the Vietnamese language.

[0120] Table 3 shows the performance of the multilingual model of this invention and a bilingual model with the same architecture on the supplementary NEWS2018 dataset. The experimental results demonstrate that, for all medium-resource and low-resource languages, the multilingual neural machine transliteration model proposed in this invention outperforms the bilingual model for the corresponding language pairs. This indicates that the multilingual neural machine transliteration model of this invention can promote interlingual transfer and alleviate the problem of insufficient corpus for medium-resource and low-resource languages.

[0121] To compare the effects of adaptive word segmentation methods on different language pairs, this invention also conducted related experiments, and the results are shown in Table 4.

[0122] Table 4. Impact of adaptive word segmentation method on model performance on the supplementary NEWS2018 dataset.

[0123]

[0124]

[0125] In Table 4:

[0126] C2C stands for character segmentation method to character segmentation method.

[0127] S2S represents a word segmentation method to a word segmentation method.

[0128] C2S represents the transition from character segmentation to sub-word segmentation.

[0129] S2C represents the transition from sub-word segmentation to character segmentation.

[0130] En→XX indicates the direction of translation from other languages ​​to English.

[0131] XX→En indicates the direction of translation from other languages ​​to English.

[0132] XX represents a collective term for other languages.

[0133] Ru represents the Russian language.

[0134] Ar represents Arabic.

[0135] Zh represents Chinese characters.

[0136] Ja represents Japanese.

[0137] "He" is a Hebrew word.

[0138] Th represents Thai.

[0139] Ko represents the Korean word.

[0140] Fa represents the Persian word.

[0141] Bn represents Bengali.

[0142] Hi is a Hindi word.

[0143] Ta represents the Tamil language.

[0144] Kn represents the Kannada language.

[0145] Vi represents the Vietnamese language.

[0146] As shown in Table 4, the adaptive word segmentation method of this invention yields different results for different languages. This invention found that for the En→XX translation direction, the "C2C" segmentation method provides better performance, while for the XX→En translation direction, "C2C" is not the best choice. This invention suggests that the "C2C" segmentation method is better for monolingual to multilingual translation, while the "C2S" method is generally superior for multilingual to monolingual translation. This may be because English tends to use sub-word segmentation, while other languages ​​tend to use character segmentation.

[0147] Furthermore, this invention observes that the segmentation scheme has a more significant impact when applied to the target language than when applied to the source language. This can be observed by comparing the performance differences between C2C and S2C versus C2C and C / S2S in the transliteration of En→XX, and between C2S and C / S2S versus C2S and C / S2C in the transliteration of XX→En. This suggests that this invention should first consider selecting a segmentation scheme suitable for the target language before adapting it to the source language.

[0148] The embodiments described above are only used to illustrate the technical ideas and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The patent scope of the present invention should not be limited by these embodiments. That is, any equivalent changes or modifications made in accordance with the spirit disclosed in the present invention still fall within the patent scope of the present invention.< / eos> < / bos>

Claims

1. A multilingual neural machine translation system, characterized in that, This system is built upon the Transformer basic model architecture based on the self-attention mechanism; This includes a multi-language pair translation model built with a unified architecture; training the system using a dataset containing multiple language pairs; segmenting sentences using a writing system and / or phonetic structure adapted to the corresponding language, and adding distinguishing symbols for the current language at the beginning of the source and target language sentences respectively; dividing multiple language pairs in the sample dataset into three groups according to the size of the training samples: high-resource language group: training samples > 98K, including Russian, Arabic, Chinese, and Japanese; medium-resource language group: training samples > 30K, including Thai, Hebrew, Korean, and Persian; low-resource language group: training samples < 15K, other languages, using a sampling strategy to balance the data distribution among different language pairs.

2. The multilingual neural machine translation system according to claim 1, characterized in that, The system includes an encoder assembly, which includes a multi-layer encoder. Each encoder layer includes a self-attention mechanism module A, a normalization module A, a feedforward neural network module A, and a normalization module B connected in sequence.

3. The multilingual neural machine translation system according to claim 2, characterized in that, The system also includes a decoder component, which comprises a multi-layer decoder. Each layer of the decoder includes a self-attention mechanism module B, a normalization module C, a cross-attention mechanism module, a normalization module D, a feedforward neural network module B, and a normalization module E connected in sequence.

4. The multilingual neural machine translation system according to claim 1, characterized in that, Translation models for multiple languages ​​are built using the fairseq framework.

5. A method for multilingual neural machine translation using the multilingual neural machine translation system according to any one of claims 1 to 4, characterized in that, Sentences are segmented using a writing system and / or phonetic structure adapted to the corresponding language, and distinguishing symbols for the current language are added at the beginning of both the source and target language sentences. Based on the size of the training samples, multiple language pairs in the dataset are divided into three groups: high-resource language group (training samples > 98K, including Russian, Arabic, Chinese, and Japanese); medium-resource language group (training samples > 30K, including Thai, Hebrew, Korean, and Persian); and low-resource language group (training samples < 15K, other languages). A sampling strategy is used to balance the data distribution among different language pairs.

6. The multilingual neural machine transliteration method according to claim 5, characterized in that, When training a multilingual neural machine translation system, the source language sentences and target language sentences used as training samples are segmented according to the word segmentation methods suitable for their respective languages ​​before being used for training.

7. The multilingual neural machine transliteration method according to claim 5, characterized in that, The multilingual neural machine translation system uses the Adam optimizer to optimize parameters during training, enabling... = 0.9, = 0.98, label smoothing rate set to 0.1; The learning rate is scaled based on the inverse square root of the number of steps, with a warm-up of 4K steps and a weight decay rate of 0.0001. During the decoding phase, beam search decoding is performed with a beam size of 5 and a length penalty of 1.

Images