A code-switching lexical overlap enhancement method for unsupervised neural machine translation

Through the code-converted vocabulary overlap enhancement method of unsupervised neural machine translation, the problem of poor performance of unsupervised translation models on long-distance language pairs is solved. Through the vocabulary overlap and text replacement technology, the translation quality and language model are significantly improved. comprehension ability.

CN114492476BActive Publication Date: 2024-11-15TIANJIN UNIV
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202210114304.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-30
Publication Date
2024-11-15
Estimated Expiration
2042-01-30

AI Technical Summary

Technical Problem

The unsupervised neural machine translation model has poor performance on long-distance language pairs and lacks vocabulary sharing information and continuous alignment signals, resulting in poor translation performance, especially difficult to achieve semantic spatial alignment between languages ​​that are not morphologically similar.

Method used

Using unsupervised neural machine translation code-transformed vocabulary overlap enhancement method, the word vectors of different languages ​​are mapped to the same space through the Vecmap tool, and the initial dictionary is generated using the CSLS distance metric, noise vocabulary is filtered, high-quality dictionaries are formed, and high-quality dictionaries are formed in text. Replace the source language vocabulary in the translation as the target language vocabulary to enhance the alignment signal in model training.

Benefits of technology

The performance of the unsupervised translation model on long-distance language pairs has been significantly improved, language similarity has been enhanced, and translation quality has been improved. Through experimental verification, significant BLEU value improvement and language model confusion have been achieved on multiple language pairs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114492476B_ABST
    Figure CN114492476B_ABST
Patent Text Reader

Abstract

The present invention discloses a code switching vocabulary overlap enhancement method for unsupervised neural machine translation, which includes three processing stages: dictionary generation, filtering and text code conversion. Step 1: dictionary generation: select the most similar words to generate an initial dictionary V init ; Step 2, filter the initial dictionary V init Noise in the text; Step 3, text code conversion: using the high-quality final dictionary model V filtered Code-switching the text, that is, completely replacing the words in the source language with the final dictionary model V filtered The corresponding words in the target language are then used in the subsequent unsupervised translation model training using the corpus after the text code conversion. Compared with the prior art, the present invention can alleviate the disadvantage of long-distance language pairs not sharing vocabulary in unsupervised translation, improve the similarity between the two languages, and thus improve the performance of the unsupervised translation model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing and proposes a classification framework for learning natural language processing tasks inspired by cognition. Background Art

[0002] Neural machine translation (NMT) has shown excellent performance in the field of machine translation in recent years and has become the mainstream method of machine translation. However, the training of neural machine translation models requires large-scale parallel corpora, which are very difficult to obtain and require expensive labor costs. In addition, for many minority languages, manual annotation of data is often not feasible. This has brought about the popularization of neural machine translation models. Heavy Big obstacle.

[0003] Unsupervised neural machine translation technology only needs to use monolingual data, which can be easily obtained through web crawlers. The amount of data far exceeds parallel corpora, and the acquisition cost is also very low. Unsupervised neural machine translation uses subword technology (such as BPE) to achieve the sharing of word vector weights for words with the same morphology in different languages, and shares the parameters of the encoder. Using existing language model pre-training methods (such as MLM), the same semantic space is implicitly used to represent semantically similar words in different languages, and the denoising task (Denoise Task) and back-translation technology (Back-Translation) are used to further improve the model performance. The performance of current unsupervised translation technology on similar language pairs, such as English-French and English-German, is not far from that of supervised translation models. Although unsupervised translation technology has made breakthrough progress in some language pairs, such as English-French and English-German, the performance of unsupervised translation technology is very low for long-distance language pairs, such as English-Chinese. One of the main features of unsupervised neural machine translation technology is that it makes good use of the similarity in word morphology between two languages, and naturally performs some vocabulary transfer and sharing. However, this advantage does not exist between languages ​​that are completely dissimilar in morphology. Unsupervised translation models between distant language pairs cannot obtain sufficient vocabulary sharing information. During the training process, there is no continuous alignment signal, and the semantic space of the two languages ​​cannot be implicitly aligned, resulting in a lack of model translation performance. In addition, distant language pairs are often accompanied by differences in vocabulary and word order, which poses great difficulties for the training of multilingual language models, resulting in worse performance of language models and poor performance of cross-language embedding representations obtained through training.

[0004] Unsupervised cross-lingual embedding assumes the isomorphism of embedding spaces and proposes to use an orthogonal linear mapping to map two monolingual embedding spaces into a shared semantic space in an unsupervised manner, providing a feasible research route for unsupervised translation models and unsupervised dictionary generation. Summary of the invention

[0005] The present invention aims to propose a code-switching vocabulary overlap enhancement method for unsupervised neural machine translation, which implements text code-switching for the most similar words between different language vocabularies, and completely replaces the words on the source language side with the corresponding words on the target language side.

[0006] The present invention is implemented by adopting the following technical solutions:.

[0007] A code-switching vocabulary overlap enhancement method for unsupervised neural machine translation includes three processing stages: dictionary generation, filtering and text code-switching. The method specifically includes the following steps:

[0008] Step 1: Dictionary generation: Use the Vecmap tool to map the monolingual word vectors obtained based on fastText training, and map the vector spaces of the two languages ​​to the same space; then use the CSLS distance measurement method to calculate the similarity between words in different languages, select the most similar words, and generate the initial dictionary V init ;

[0009] Step 2: Filter the initial dictionary V init Noise present in

[0010] Step 3: Text Code Switching: Using the High-Quality Final Dictionary Model V filtered Code-switching the text, that is, completely replacing the words in the source language with the final dictionary model V filtered The corresponding target language words in the text are then used in the unsupervised translation model training.

[0011] Compared with the existing unsupervised neural machine translation technology, the present invention has the following advantages:

[0012] Without changing the original algorithm, the training data is preprocessed and words with the same semantics are replaced. This allows the unsupervised translation model to continuously obtain alignment signals between two distant language pairs during training, which helps the semantic spaces of the two languages ​​to get closer during training and improve the similarity between the two languages, thereby improving the performance of the unsupervised translation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of the code switching vocabulary overlap enhancement method for unsupervised neural machine translation of the present invention. ;

[0014] Figure 2 This is a schematic diagram of the PCA visualization analysis results of the embedding layer parameters;

[0015] Figure 3It is a graph of language model perplexity;

[0016] Figure 4 This is a comparison diagram of Zero-Shot machine translation results. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0018] The present invention discloses a code-switching lexical overlap enhancement method (Code-Switching Lexical Overlap Enhancement, CLOE) for unsupervised neural machine translation. The CLOE algorithm of the present invention aims to solve the problem of low similarity of long-distance language pairs and poor translation model performance in unsupervised neural machine translation. Specifically, in unsupervised translation, long-distance language pairs lack morphologically similar shared vocabulary, which makes it difficult for the unsupervised translation model to map the semantic spaces of two dissimilar languages ​​into the same space, greatly reducing the translation performance of unsupervised machine translation in this context. The CLOE algorithm of the present invention is proposed to alleviate the disadvantage of long-distance language pairs having no shared vocabulary in unsupervised translation, improve the similarity of the two languages, and thus improve the performance of the unsupervised translation model.

[0019] like Figure 1 As shown, it is a flow chart of the code conversion vocabulary overlap enhancement method of unsupervised neural machine translation of the present invention. The overall process of this method is divided into three parts, namely dictionary generation, filtering and text code conversion. Replace the source language vocabulary with x, replace the target language vocabulary with y, use the encoder-decoder architecture based on the transformer model, input the source sentence as {x1, x2, x3, x4, x5}, and the corresponding target translation sentence is represented as {y5, y6, y7, y8, y9}. Use the Vecmap tool to perform word-level alignment to obtain a bilingual dictionary. According to the threshold filtering and mutual matching principles, the dictionary quality is improved and the number of incorrect word alignments is reduced. After that, replacement is performed based on a high-quality bilingual dictionary to obtain a sentence of the form {x1, y6, y7, x4, x5}, traverse all source sentences, and obtain code conversion corpus. Unsupervised translation training is performed based on the code conversion corpus. During the model training process, sentences from different languages ​​are processed simultaneously to enhance the bilingual similarity, thereby improving the model performance. The specific steps are as follows:

[0020] Step 1: Dictionary generation. Since CLOE enhances unsupervised translation, it does not provide an artificially generated bilingual dictionary. This method uses the Vecmap tool to map the monolingual word vectors obtained based on fastText training, mapping the vector spaces of the two languages ​​to the same space; next, the CSLS distance measurement method is used to calculate the similarity between words in different languages, select the most similar words, and generate the initial dictionary V init ;

[0021] Step 2: Filter. Initial dictionary V init There is a lot of noise in the dictionary; two filtering principles are designed to filter the dictionary:

[0022] Step 2.1: Perform threshold filtering based on the hyperparameter τ to obtain the filter dictionary model V thres , which is expressed as follows:

[0023] V thres ={(w s ,w t )|CSLS(w s ,w t )>τ,(w s ,w t )∈V init

[0024] Among them, w t represents the target vocabulary, w s Represents the source vocabulary, and calculates the similarity based on the fastText word vector corresponding to the vocabulary, such as calculating the vector similarity based on L2 distance and cosine similarity. This method calculates the similarity of vocabulary based on CSLS, which is a margin-based vector distance metric. Compared with cosine similarity, it can effectively alleviate the hubness problem in high-dimensional space and improve the accuracy of similarity calculation. Determine the quality of the dictionary in the dictionary: the larger the τ value, the fewer word pairs that meet the requirements in the dictionary, and the higher the accuracy of the dictionary. In principle, τ is set to 0.3, and it can be adjusted between 0.25 and 0.35 in actual application.

[0025] Step 2.2, optimal mutual matching. After generating the initial dictionary V of the two languages init and filter dictionary V thresIn the process of translation, the dictionary model is established independently, so two dictionaries with opposite translation directions are obtained. The optimal mutual matching principle will combine the word pairs in the two dictionaries and only retain the word alignment results that exist in both dictionaries. For example, if there is a word s in the source language that is aligned in the direction of "source language -> target language" to get the most similar word t in the target language, and the word t is aligned in the direction of "target language -> source language" is also word s, then word s and word t are a set of mutual matches, and the final dictionary model V filtered This alignment is preserved in .

[0026] Step 3: Text code conversion. Using the high-quality final dictionary model V filtered Code-switching the text, that is, completely replacing the words in the source language with the final dictionary model V filtered The corresponding target language word in . In the subsequent unsupervised translation model training, this corpus after text code conversion is used.

[0027] The translation performance test of the code-switching vocabulary overlap enhancement method for unsupervised neural machine translation proposed in this invention was carried out through a number of analytical experiments, mainly including model embedding layer parameter visualization, language model perplexity (PPL) experiment, and language model zero-shot language. The specific description is as follows:

[0028] Experiments were conducted on multiple long-distance language pairs, mainly including English-Chinese, English-Estonian, and English-Turkish. The three languages ​​belong to different language families, and there is sufficient monolingual data for model training. Table 1 shows the experimental results of the BLEU value of the machine translation evaluation indicator. ++ means that the experimental results passed the significance test (p<0.05).

[0029] Method En→Et Et→En En→Zh Zh→En En→Tr Tr→En XLM 7.7 11.4 7.6 5.1 6.7 6.8 MASS 6.0 7.2 4.9 3.5 5.2 5.4 XLM+CLOE 8.6++ 12.6++ 8.3++ 6.0++ 7.1++ 7.9++ MASS+CLOE 7.6++ 8.8++ 6.1++ 4.7++ 5.8++ 6.3++

[0030] The experimental results in Table 1 show that the CLOE algorithm of the present invention has achieved significant improvements on two unsupervised neural machine translation baseline models XLM and MASS.

[0031] The present invention conducts specific translation experiments in English-Estonian, English-Chinese and English-Turkish, using monolingual data from WMT2014 and newstest as the test set. The test results are shown in Table 1. XLM and MASS are the baseline models of the existing unsupervised translation models.

[0032] The present invention has achieved great improvements on both baselines with excellent performance, achieving BLEU improvements of 0.4 to 0.9 and 0.6 to 1.6 in the En-X direction, and 0.9 to 1.2 and 0.9 to 1.6 in the X-EN direction. After significance tests, with a P-value of 0.05, all results passed the significance test, indicating that the present invention can significantly improve the performance of the unsupervised translation model.

[0033] 1. Visual analysis of the model embedding layer: Figure 2 As shown, it is a schematic diagram of the PCA visualization analysis results of the embedding layer parameters. The unsupervised translation model of English-Chinese is selected as the object of the analysis experiment, the parameters of the embedding layer of the translation model are derived, and PCA visualization is used for it. It can be seen that in the embedding layer of the language model trained based on the original corpus, there are very few shared vocabularies 100 between the two languages, and the source vocabulary 200 is far away from the target language vocabulary 300, and there is no obvious overlap between the two, indicating that the similarity between the two languages ​​is not high. The data processed by the CLOE algorithm of the present invention can greatly enhance the overlap of the two languages ​​through the same language model training method, which shows that it is significantly helpful to improve the language similarity.

[0034] 2. Language Model Perplexity (PPL) Experiment,For the English-Chinese language pair, we record its perplexity during the language model training process, such as Figure 3 As shown, it is a language model perplexity curve. It can be seen that the language model with the CLOE algorithm of the present invention can maintain a consistent lower perplexity, indicating that the CLOE algorithm of the present invention can significantly enhance the similarity of two distant language pairs and help the language model better understand the distant language.

[0035] 3. Zero-Shot Translation Experiment

[0036] The unsupervised translation task is a text generation task. We also analyze the text generation ability of the language model under this method. Based on the language model MASS, we examine the impact of the original corpus and the corpus processed by the CLOE algorithm of the present invention on the zero-shot ability of text translation, that is, the language model is directly used for the translation task. Figure 4As shown, it is a schematic diagram of the comparison of Zero-Shot machine translation results. It can be seen that in the two application scenarios of English-Chinese (en2zh) and Chinese-English (zh2en), compared with the machine translation scenario using only MASS (MASS, natural language generation with pre-training mechanism), MASS+CLOE, i.e., the CLOE algorithm of the present invention is combined with MASS, can continuously improve the zero-shot translation performance during the training of the language model, indicating that during the training of the language model, the model not only constructs the language structure of a single language, but also can interact and share with another language, so that the bilingual language model can obtain further language understanding, and the natural translation ability of the two languages ​​is improved.

[0037] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A code-switching vocabulary overlap enhancement method for unsupervised neural machine translation, characterized in that: The method includes three processing stages: dictionary generation, filtering and text code conversion. The method specifically includes the following steps: Step 1: Dictionary generation: Use the Vecmap tool to map the monolingual word vectors obtained based on fastText training, and map the vector spaces of the two languages ​​to the same space; then use the CSLS distance measurement method to calculate the similarity between words in different languages, select the most similar words, and generate the initial dictionary V init ; Step 2: Filter the initial dictionary V init The filtering adopts the combination of the following two principles: Step 2.1, threshold filtering principle: threshold filtering is performed based on the hyperparameter τ to obtain the filtering dictionary model V thres , which is expressed as follows: V thres ={(in s ,In t )|CSLS(in s ,In t )>τ,(in s ,In t )∈V init Among them, w t represents the target vocabulary, w s Represents source vocabulary; The quality of the dictionary in the dictionary is determined according to the similarity metric of CSLS; the larger the value of the hyperparameter τ, the fewer the number of word pairs that meet the requirements in the dictionary, and the higher the accuracy of the dictionary; Step 2.2, Optimal Mutual Matching Principle: When generating the initial dictionary V of the two languages init and filter dictionary v thres In the process, the dictionary models are established independently, thus obtaining two dictionary models with opposite translation directions. The optimal mutual matching principle combines the word pairs in the two dictionary models and only retains the word alignment results that exist in both dictionaries. Step 3: Text Code Switching: Using the High-Quality Final Dictionary Model V filtered Code-switching the text, that is, completely replacing the words in the source language with the final dictionary model V filtered The corresponding target language words in the text are then used in the unsupervised translation model training.

Citation Information

Patent Citations

  • Pre-training method and device of intelligent translation model and storage medium

    CN111460838A

  • Chinese- Vietnamese unsupervised neural machine translation method fusing EMD minimized bilingual dictionary

    CN111753557A

  • Translation quality evaluation method, system and device based on unsupervised learning and medium

    CN111859992A

  • High-quality Mongolian-Chinese unsupervised neural machine translation method

    CN112580373A

  • Bilingual word alignment method and system

    CN113591496A