Cross-language translation model training method based on transfer learning
Patent Information
- Application Number
- CN202411459381.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2026-10-20
AI Technical Summary
但同样存在不少低资源语言，可用的平行语料资源有限，若要构建高质量的翻译模型，小规模的平行语料资源难以实现
[0025]1.本方案通过源语言与枢轴语言(表示选定的高资源语言，与目标语言在语法结构上越接近越好，例如英语与南非语，中文作为源语言，英语属于高资源语言，南非语则为低资源语言)的平行语料库，对机器翻译模型进行预训练，得到预训练完成的初级翻译模型。然后基于源语言与目标语言的平行语料库，构建源语言与目标语言的伪平行语料库，再利用伪平行语料库对初级翻译模型进行调节训练，得到训练完成的跨语言翻译模型。本方案利用迁移学习的思想，迁移学习是一种强大的机器学习方法，其核心思想是将在一个任务(源任务)上学到的知识或模型迁移到另一个相关任务(目标任务)上，以提高新任务的性能。这种方法充分利用了已有的数据和模型资源，减少了目标任务对大量新数据的依赖，加快了模型训练速度，并提高了模型的泛化能力。考虑到跨语言翻译任务中，即便是低资源的小语种，通常也有与之语法结构较为接近的高资源语言，因此，将迁移学习的思想应用到低资源语言的机器翻译任务中，能够起到不错的效果。虽然迁移学习能够减少对大量新数据的依赖，但在实际应用中，更多的数据仍然有助于进一步提高翻译准确性，更多的数据可以提供更丰富的语言特征和上下文信息，使模型能够更好地理解源语言并生成更准确的目标语言。当目标领域的数据非常有限时，使用伪平行语料库可以帮助模型更好地适应目标领域的语言风格和词汇使用。因此本方案利用源语言与目标语言之间已有的小规模的平行语料库，进行数据扩充，构建伪平行语料库，这种扩充不同于简单的数据增强，能够很好地考虑到词的含义和语法结构，据此扩充源语言与目标语言的可用数据，从而能够进一步提升翻译质量。
Smart Images
Figure BDA0005091434530000041 
Figure BDA0005091434530000043 
Figure BDA0005091434530000052
Abstract
Description
Technical Field
[0001] This application relates to the field of machine translation technology, and more specifically, to a method for training a cross-language translation model based on transfer learning. Background Technology
[0002] In machine translation, for high-resource languages (languages with abundant available corpora between the source and source languages, including but not limited to text datasets, bilingual parallel corpora, labeled data, and computational resources), large-scale bilingual parallel corpora are readily available, which is beneficial for training machine translation models with extremely high translation accuracy, thereby achieving intelligent translation. However, there are also many low-resource languages where available parallel corpora are limited, making it difficult to build high-quality translation models with small-scale parallel corpora.
[0003] Currently, traditional data augmentation methods, while expanding the training corpus to some extent to address the issues of small data scale and scarcity in low-resource languages, struggle to fully exploit the inherent semantic connections and syntactic structural features of the language. This results in less than ideal data augmentation effects and difficulty in improving the translation accuracy of the trained model. Therefore, effectively improving translation quality in machine translation tasks for low-resource languages is a key technical problem that needs to be solved in this field. Summary of the Invention
[0004] The purpose of this application is to provide a cross-language translation model training method based on transfer learning. This method selects a pivotal language (which needs to be a high-resource language) with a similar grammatical structure to the target language using the idea of transfer learning. The machine translation model is pre-trained using a parallel corpus. Then, a pseudo-parallel corpus of the source and target languages is constructed to expand the data scale of the target language. The expanded pseudo-parallel corpus (which also includes existing parallel corpora) is used to adjust the initial translation model, resulting in a well-trained cross-language translation model. This effectively improves the quality and accuracy of machine translation for low-resource languages.
[0005] To achieve the above objectives, the embodiments of this application are implemented in the following manner:
[0006] In a first aspect, embodiments of this application provide a method for training a cross-language translation model based on transfer learning, comprising: pre-training a machine translation model using parallel corpora of the source language and the pivot language to obtain a pre-trained primary translation model, wherein the pivot language represents a selected high-resource language; constructing a pseudo-parallel corpus of the source language and the target language based on the parallel corpora of the source language and the target language, wherein the source language represents the language to be translated and the target language represents the language to which it is to be translated; and adjusting and training the primary translation model using the pseudo-parallel corpus to obtain a trained cross-language translation model.
[0007] In conjunction with the first aspect, in the first possible implementation of the first aspect, a pseudo-parallel corpus of the source and target languages is constructed based on parallel corpora of the source and target languages. This includes: acquiring monolingual corpus datasets of the source language and the target language; using unsupervised learning algorithms to perform unsupervised word embedding pre-training on the monolingual corpus of the source language based on the monolingual corpus dataset of the source language, learning the vector representation of each word, and capturing the semantic relationships between words to obtain the word vector of each word in the source language, forming a word vector set of the source language; and using the monolingual corpus of the target language... The dataset employs unsupervised learning algorithms for unsupervised word embedding pre-training on monolingual corpora of the target language. This learns the vector representation of each word and captures the semantic relationships between words, resulting in word vectors for each target language word, forming a target language word vector set. Based on parallel corpora of the source and target languages, the source language word vector set, the target language word vector set, the source language monolingual corpus dataset, and the target language monolingual corpus dataset, an auxiliary translation model is trained. Based on the auxiliary translation model, the source language monolingual corpus and the corresponding target language translation corpus are determined, constructing a pseudo-parallel corpus.
[0008] In conjunction with the first possible implementation of the first aspect, in the second possible implementation of the first aspect, an auxiliary translation model is trained based on parallel corpora of the source and target languages, word vector sets of the source and target languages, monolingual corpora of the source and target languages, and monolingual corpora of the target language. This includes: determining semantically corresponding word pairs between the source and target languages based on the parallel corpora of the source and target languages; determining a word embedding space mapping matrix from the word embedding space of the source language to the word embedding space of the target language based on the semantically corresponding word pairs between the source and target languages; and initializing and training the auxiliary translation model based on the word embedding space mapping matrix, word vector sets of the source and target languages, monolingual corpora of the source and target languages to obtain the trained auxiliary translation model.
[0009] In conjunction with the second possible implementation of the first aspect, in the third possible implementation of the first aspect, an auxiliary translation model is initialized and trained based on the word embedding space mapping matrix, the source language word vector set, the target language word vector set, the source language monolingual corpus dataset, and the target language monolingual corpus dataset to obtain the trained auxiliary translation model. This includes: initializing the auxiliary translation model based on the word embedding space mapping matrix, the source language word vector set, and the target language word vector set; determining parallel corpus data for the source and target languages based on parallel corpora of the source and target languages, and dividing the parallel corpus data into a parallel corpus training set and a parallel corpus validation set according to a set ratio; and determining a portion of the source language monolingual corpus dataset. The source language corpus data is divided into a training set and a validation set according to a set ratio. The auxiliary translation model is pre-trained using the parallel training set, and the word embedding space mapping matrix is updated accordingly. The source language training set is then merged into the parallel training set to form a new corpus training set, which is used to iteratively train the auxiliary translation model, updating the word embedding space mapping matrix again. The parallel validation set and the source language validation set are combined into a validation set, which is used to validate the auxiliary translation model. If the validation is successful, the training of the auxiliary translation model is complete; otherwise, the model is retrained using the original training set and the new corpus training set to obtain the trained auxiliary translation model.
[0010] In conjunction with the third possible implementation of the first aspect, the fourth possible implementation of the first aspect involves merging the source language corpus training set into a parallel corpus training set to form a new corpus training set. This is used to iteratively train the auxiliary translation model and update the word embedding space mapping matrix. The process includes: merging the source language corpus training set into a parallel corpus training set; inputting each training data point in the corpus training set into the auxiliary translation model to obtain the corresponding literal translation corpus data, then translating the literal translation corpus data back into the source language to obtain the back-translated corpus data corresponding to the training data; calculating the similarity based on the training data, the literal translation corpus data, and the corresponding back-translated corpus data to obtain the similarity value corresponding to the training data, and obtaining human evaluation; and updating the word embedding space mapping matrix based on the similarity value and human evaluation.
[0011] In conjunction with the fourth possible implementation of the first aspect, in the fifth possible implementation of the first aspect, similarity calculation is performed based on training data, literal translation corpus data, and corresponding back-translation corpus data to obtain the similarity value corresponding to the training data. This includes: determining the word vector of each word in the training data, determining the word vector of each word in the literal translation corpus data, and determining the word vector of each word in the back-translation corpus data; determining the correspondence between the word vectors of the training data, the word vectors of the literal translation corpus data, and the word vectors of the back-translation corpus data; and calculating the similarity value corresponding to the training data based on the word vectors of the corresponding training data, the word vectors of the literal translation corpus data, and the word vectors of the back-translation corpus data.
[0012] In conjunction with the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, based on the word vectors of the corresponding training data, the word vectors of the literal translation corpus, and the word vectors of the back-translation corpus, the similarity value corresponding to the training data is calculated, including: calculating the similarity component between the training data and the corresponding literal translation corpus using the following formula:
[0013]
[0014] in, Let w be the similarity component between the i-th training data and the corresponding translated corpus, where n is the total number of word vectors in that training data, and w is the similarity component between the i-th training data and the corresponding translated corpus. ij Let v be the word vector of the j-th word in the training data. ij For the direct translation corpus data and word vectors w ij The corresponding word vector, w ij ·v ij Word vector w ij With word vector v ij The dot product, ||w ij ||and||v ij || These are the word vectors w ij and word vector v ij The similarity component between the literal translation corpus and the corresponding back-translation corpus is calculated using the following formula:
[0015]
[0016] in, Let v be the similarity component between the literal translation corpus and the back-translation corpus corresponding to the i-th training data, n be the total number of word vectors in the literal translation corpus, and v be the similarity component between the literal translation corpus and the back-translation corpus. ij The word vector w in the literal translation corpus and the training data is... ij The corresponding word vector, u ij For back-translation of corpus data and word vector v ij The corresponding word vector, vij ·u ij Word vector v ij With word vector u ij The dot product, ||v ij || and ||u ij || These are the word vectors v ij and word vector u ij The modulus length; the similarity components between the training data and the corresponding back-translated corpus data are calculated using the following formula:
[0017]
[0018] in, Let w be the similarity component between the i-th training data and the corresponding back-translated corpus data, n be the total number of word vectors in that training data, and w be the similarity component between the i-th training data and the corresponding back-translated corpus data. ij Let u be the word vector of the j-th word in the training data. ij For back-translation corpus data and word vector w ij The corresponding word vector, w ij ·u ij Word vector w ij With word vector u ij The dot product, ||w ij || and ||u ij || These are the word vectors w ij and word vector u ij Modulus length; based on similarity components Similarity components and similarity components Calculate the similarity value corresponding to the training data.
[0019] Combining the sixth possible implementation of the first aspect, in the seventh possible implementation of the first aspect, based on similarity components... Similarity components and similarity components Calculate the similarity value between the training data, including using the following formula:
[0020]
[0021] Among them, S i Let be the similarity value corresponding to the i-th training data, where α, β, and γ are all weight parameters.
[0022] In conjunction with the sixth possible implementation of the first aspect, in the eighth possible implementation of the first aspect, human evaluation reveals whether the meaning expressed by the training data and the corresponding back-translated corpus data is consistent. Based on the similarity value and the human evaluation, the word embedding space mapping matrix is updated, including: based on the human evaluation, determining the update range and update direction of the word embedding space mapping matrix for this update, wherein the update range is used to reveal the value of the word embedding space mapping matrix for this update, and the update direction is used to determine whether to increase or decrease the value within the update range; based on the similarity value, determining the element update amount within the update range of the word embedding space mapping matrix for this update; based on the element update amount, using the gradient descent method to update the value within the update range along the update direction.
[0023] In conjunction with the first possible implementation of the first aspect, in the ninth possible implementation of the first aspect, based on the auxiliary translation model, the source language monolingual corpus and the corresponding target language translation corpus are determined, and a pseudo-parallel corpus is constructed. This includes: inputting the remaining source language corpus data from the source language monolingual corpus dataset into the auxiliary translation model one by one; translating the input source language corpus data by the auxiliary translation model and outputting the corresponding target language translation corpus data; and using each source language corpus data and the corresponding target language translation corpus data as a set of pseudo-parallel corpus data, thereby constructing a pseudo-parallel corpus.
[0024] Beneficial effects:
[0025] 1. This approach pre-trains a machine translation model using parallel corpora of the source language and a pivot language (representing a selected high-resource language, the closer to the target language in grammatical structure, the better; for example, English and Afrikaans, with Chinese as the source language, English being a high-resource language and Afrikaans a low-resource language). This results in a pre-trained, basic translation model. Then, based on the parallel corpora of the source and target languages, a pseudo-parallel corpus of the source and target languages is constructed. This pseudo-parallel corpus is then used to fine-tune and train the basic translation model, resulting in a fully trained cross-language translation model. This approach utilizes the concept of transfer learning, a powerful machine learning method. Its core idea is to transfer knowledge or models learned on one task (the source task) to another related task (the target task) to improve performance on the new task. This method fully utilizes existing data and model resources, reduces the target task's dependence on large amounts of new data, accelerates model training, and improves the model's generalization ability. Considering that even low-resource languages in cross-language translation tasks often have high-resource languages with similar grammatical structures, applying transfer learning to machine translation tasks of low-resource languages can yield good results. While transfer learning can reduce reliance on large amounts of new data, in practice, more data still helps improve translation accuracy. More data provides richer linguistic features and contextual information, enabling the model to better understand the source language and generate more accurate target language. When data in the target domain is very limited, using pseudo-parallel corpora can help the model better adapt to the language style and vocabulary usage of the target domain. Therefore, this approach utilizes existing small-scale parallel corpora between the source and target languages to augment the data and construct pseudo-parallel corpora. This augmentation differs from simple data augmentation; it takes into account word meaning and grammatical structure, thereby expanding the available data in both the source and target languages and further improving translation quality.
[0026] 2. Obtain monolingual corpora of the source language and the target language. Based on the monolingual corpora of the source language, use unsupervised learning algorithms (such as FastText, Word2Vec, etc.) to perform unsupervised word embedding pre-training on the monolingual corpora of the source language, learn the vector representation of each word, and capture the semantic relationships between words to obtain the word vector of each word in the source language, forming a word vector set of the source language. Based on the monolingual corpora of the target language, use unsupervised learning algorithms (such as FastText, Word2Vec, etc., but the unsupervised learning algorithm must be consistent with that used for the monolingual corpora of the source language) to perform unsupervised word embedding pre-training on the monolingual corpora of the target language, learn the vector representation of each word, and capture the semantic relationships between words to obtain the word vector of each word in the target language, forming a word vector set of the target language. Based on parallel corpora of the source and target languages, word vector sets of the source and target languages, and monolingual corpora of the source and target languages, an auxiliary translation model is trained. Based on this auxiliary translation model, the source language monolingual corpora and their corresponding target language translation corpora are determined, thus constructing a pseudo-parallel corpus. An auxiliary translation model is introduced to generate pseudo-translation corpora during the construction of the pseudo-parallel corpus. The purpose of this auxiliary translation model is to translate the source language monolingual corpora into the target language, thereby generating pseudo-parallel corpus pairs. The auxiliary translation model (used to generate pseudo-parallel corpus data) differs from the primary translation model (which serves as the starting point for transfer learning). The auxiliary translation model used to construct the pseudo-parallel corpus is specifically trained for this task. Of course, if an existing model is available, it can be used directly, for example, in scenarios where the source language is extended to the target language, such as in more refined, sub-domain translation tasks. The reason for using a pseudo-parallel corpus to adjust the pre-trained primary translation model is that it provides a solution for expanding training data under limited resources. By introducing pseudo-parallel corpora, the generalization ability of the model can be increased, enabling it to better handle unseen data. At the same time, the adjustment of the primary translation model by the pseudo-parallel corpora can achieve domain adaptation, thereby improving the domain adaptability of the cross-language translation model.
[0027] 3. By using parallel corpora of the source and target languages, semantically corresponding word pairs between the two languages are identified. Furthermore, a word embedding space mapping matrix is determined, from the source language word embedding space to the target language word embedding space. Based on this mapping matrix, the source language word vector set, the target language word vector set, the source language monolingual corpus dataset, and the target language monolingual corpus dataset, the auxiliary translation model is initialized and trained, resulting in the trained auxiliary translation model. During initialization and training, this approach uses a portion of the parallel corpora of the source and target languages, as well as the source language monolingual corpus data, as training data. This addresses the sample balance issue, improves training efficiency, and avoids various problems caused by sample imbalance during the training phase of the auxiliary translation model. When training the translation assistance model using training data, the training scheme was also adjusted. The translation assistance model was pre-trained using a parallel corpus training set, and the word embedding space mapping matrix was updated accordingly. Then, the source language corpus training set was added to the parallel corpus training set and merged into a corpus training set. The translation assistance model was then trained iteratively, and the word embedding space mapping matrix was updated accordingly. The parallel corpus validation set and the source language corpus validation set were then merged into a corpus validation set, and the translation assistance model was validated using the corpus validation set until the training objective was achieved (for example, if the validation did not meet the target, a second training was required). Furthermore, in the process of training and updating the word embedding space mapping matrix, not only is similarity calculated, but also human evaluation is introduced (human evaluation reveals whether the meaning expressed by the training data and the corresponding back-translated corpus data is consistent), thereby determining the update range and direction of the updated word embedding space mapping matrix. The similarity value is used to determine the number of elements updated within the update range of the updated word embedding space mapping matrix, and a gradient descent strategy is adopted to update the word embedding space mapping matrix in order to train an auxiliary translation model with high translation accuracy, which can be used to generate pseudo-parallel corpus data and construct a pseudo-parallel corpus.
[0028] 4. In the similarity value calculation process, unlike the traditional similarity measurement model of source language corpus data - direct translation corpus data, a similarity measurement model of source language corpus data (i.e., training data), direct translation corpus data, and back translation corpus data is introduced. A similarity measurement scheme is designed, and the allocation of weights for different similarity components is considered, thus taking into account the similarity components well. ( (Similarity components between the i-th training data and the corresponding translated corpus data) ( (These are the similarity components between the literal translation corpus and the corresponding back-translation corpus data corresponding to the i-th training data) and similarity components. ( The similarity components between the i-th training data and the corresponding back-translated corpus data are analyzed to determine the relationship between them and the influence of the component size on the weights, thereby more accurately determining the similarity value. This information is then used to update the word embedding space mapping matrix, improve the translation quality of the auxiliary translation model, and ultimately improve the quality of the pseudo-parallel corpus data.
[0029] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 A flowchart illustrating a cross-language translation model training method based on transfer learning, provided in an embodiment of this application.
[0032] Figure 2 A flowchart for constructing a pseudo-parallel corpus of the source and target languages. Detailed Implementation
[0033] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0034] Please see Figure 1 , Figure 1 This is a flowchart of a cross-language translation model training method based on transfer learning. In this embodiment, the cross-language translation model training method based on transfer learning may include steps S10, S20, and S30.
[0035] Since this approach utilizes the concept of transfer learning, the first step is to determine a suitable pivot language for pre-training the machine translation model. The pivot language is chosen with the target language (the language to be translated) as a reference, selecting a high-resource language with a grammatical structure closer to the target language. This ensures the effectiveness of transfer learning. For example, suppose the source language (the language to be translated) is Chinese, and the target language is Afrikaans. English is a high-resource language for the source language (Chinese), possessing abundant available resources (such as text datasets, bilingual parallel corpora, labeled data, and computational resources). Furthermore, Afrikaans is a variant of Dutch, further simplifying its grammatical structure and sharing many similarities with English in vocabulary and sentence structure. Therefore, English can be chosen as the pivot language.
[0036] Once the pivot language is determined, step S10 can be executed.
[0037] Step S10: Use parallel corpora of the source language and the pivot language to pre-train the machine translation model to obtain a pre-trained primary translation model, where the pivot language represents the selected high-resource language.
[0038] In this embodiment, parallel corpora of the source language and the pivot language can be collected to pre-train a machine translation model (such as the Transformer model or its variants, the encoder-decoder framework of a recurrent neural network (RNN), etc.; this embodiment uses the encoder-decoder framework of an RNN as an example) to obtain a pre-trained primary translation model. For example, readily available parallel corpora can be obtained, or sufficient parallel corpora can be obtained by collecting data (utilizing publicly available resources on the internet, such as UN documents, EU translation databases, etc., or scraping parallel corpora from suitable bilingual web pages) and processing it to obtain a large-scale parallel corpus for pre-training the machine model to obtain a primary translation model. This process is already well-documented, and the improvements in this solution do not lie in this step; therefore, they will not be elaborated upon here.
[0039] After obtaining the pre-trained primary translation model, step S20 can be run.
[0040] Step S20: Based on the parallel corpora of the source language and the target language, construct a pseudo-parallel corpus of the source language and the target language, where the source language represents the language that needs to be translated and the target language represents the language that is to be translated into.
[0041] To improve the model's generalization and domain adaptability, it is necessary to expand the scale of the parallel corpus of the target language. Therefore, this embodiment uses the existing small-scale parallel corpus of the source language and the target language as the basis for constructing a pseudo-parallel corpus of the source language and the target language, thereby expanding the scale of the parallel corpus.
[0042] In this embodiment, the process of constructing a pseudo-parallel corpus of the source language and the target language is as follows: Figure 2 As shown, it may include steps S21, S22, S23, S24, and S25.
[0043] First, step S21 needs to be run.
[0044] Step S21: Obtain the monolingual corpus dataset of the source language and the monolingual corpus dataset of the target language.
[0045] In this embodiment, monolingual corpus datasets for the source language and the target language can be obtained. Specifically, a large number of monolingual corpora can be crawled from the internet, such as articles from news websites, blogs, forums, etc. Then, a series of preprocessing steps are performed, such as data cleaning, HTML tag removal, sentence segmentation, and character processing. This yields the monolingual corpus datasets for the source language and the target language.
[0046] After obtaining the monolingual corpus datasets of the source language and the target language, step S21 can be run.
[0047] Step S21: Based on the monolingual corpus dataset of the source language, an unsupervised learning algorithm is used to perform unsupervised word embedding pre-training on the monolingual corpus of the source language, learn the vector representation of each word, capture the semantic relationship between words, obtain the word vector of each word in the source language, and form the word vector set of the source language.
[0048] In this embodiment, based on the monolingual corpus dataset of the source language, an unsupervised learning algorithm (such as FastText, Word2Vec, etc., with FastText as an example in this embodiment) can be used to perform unsupervised word embedding pre-training on the monolingual corpus of the source language, learn the vector representation of each word, capture the semantic relationship between words, obtain the word vector of each word in the source language, and form the word vector set of the source language.
[0049] The same principle applies to processing monolingual corpus datasets of the target language; step S23 can be executed.
[0050] Step S23: Based on the target language monolingual corpus dataset, an unsupervised learning algorithm is used to perform unsupervised word embedding pre-training on the target language monolingual corpus, learn the vector representation of each word, capture the semantic relationship between words, obtain the word vector of each target language word, and form the target language word vector set.
[0051] In this embodiment, unsupervised word embedding pre-training can be performed on the monolingual corpus of the target language based on the monolingual corpus dataset of the target language, using an unsupervised learning algorithm (such as FastText, Word2Vec, etc., but it needs to be consistent with the unsupervised learning algorithm of the monolingual corpus of the source language), to learn the vector representation of each word and capture the semantic relationship between words, thereby obtaining the word vector of each word in the target language and forming a target language word vector set.
[0052] Then, step S24 can be run.
[0053] Step S24: Train the auxiliary translation model based on parallel corpora of the source and target languages, word vector sets of the source language and the target language, monolingual corpus datasets of the source language and the target language.
[0054] In this embodiment, semantically corresponding word pairs between the source and target languages can be determined based on parallel corpora of the source and target languages. Then, based on these semantically corresponding word pairs, a word embedding space mapping matrix from the source language word embedding space to the target language word embedding space is determined.
[0055] For example, based on the semantically corresponding word pairs between the source and target languages, and their respective word embedding vector spaces, a mapping matrix can be determined (e.g., by matrix factorization, machine learning algorithms, etc., which can be optimized later, as will be discussed in detail below). This mapping matrix can map the vectors in the word embedding space of the source language to the corresponding vectors in the word embedding space of the target language.
[0056] Then, based on the word embedding space mapping matrix, the source language word vector set, the target language word vector set, the source language monolingual corpus dataset, and the target language monolingual corpus dataset, the auxiliary translation model can be initialized and trained to obtain the trained auxiliary translation model.
[0057] For example, the auxiliary translation model can be initialized based on the word embedding space mapping matrix, the source language word vector set, and the target language word vector set. For instance, when initializing the auxiliary translation model (using a recurrent neural network encoder-decoder framework and setting necessary model parameters such as learning rate, batch size, number of iterations, etc.), the source language word vector set can be used to initialize the input layer of the encoder, the target language word vector set can be used to initialize the output layer of the decoder, and the word embedding space mapping matrix can be used to adjust the parameters of the encoder and decoder.
[0058] After initializing the assisted translation model, parallel corpus data for the source and target languages can be determined based on parallel corpora of the source and target languages. The parallel corpus data can then be divided into a parallel corpus training set and a parallel corpus validation set according to a set ratio (e.g., 8:2). Additionally, a portion of the source language corpus data (the data size does not exceed 3 times the parallel corpus data of the source and target languages; in this embodiment, 2 times is used as an example) can be determined from the monolingual corpus dataset of the source language. This portion of the source language corpus data can then be divided into a source language corpus training set and a source language corpus validation set according to a set ratio (e.g., 8:2).
[0059] Then, the parallel corpus training set can be used for pre-training of the translation aid model, and the word embedding space mapping matrix can be updated accordingly. Using the parallel corpus training set for pre-training here is beneficial for quickly optimizing model parameters (by updating the word embedding space mapping matrix and then feeding it back into the parameter adjustment of the translation aid model), similar to finding a benchmark, which is then used as the basis for subsequent optimization.
[0060] Subsequently, the source language corpus training set can be added to the parallel corpus training set to merge them into a corpus training set, and the auxiliary translation model can be iteratively trained, with feedback used to update the word embedding space mapping matrix.
[0061] Specifically, the source language corpus training set can be added to a parallel corpus training set to form a new corpus training set. For each training data point in the corpus training set, it is input into the auxiliary translation model to obtain the corresponding literal translation corpus data. Then, the literal translation corpus data is translated back into the source language to obtain the back-translated corpus data corresponding to the training data.
[0062] After obtaining the literal translation corpus and the corresponding back-translation corpus corresponding to the training data, similarity can be calculated based on the training data, the literal translation corpus and the corresponding back-translation corpus to obtain the similarity value corresponding to the training data and obtain human evaluation.
[0063] For example, the word vectors of each word in the training data, the word vectors of each word in the literal translation corpus, and the word vectors of each word in the back-translation corpus can be determined. Then, the correspondence between the word vectors of the training data, the word vectors of the literal translation corpus, and the word vectors of the back-translation corpus can be determined. Then, based on the corresponding word vectors of the training data, the word vectors of the literal translation corpus, and the word vectors of the back-translation corpus, the similarity value of the corresponding training data can be calculated.
[0064] For each piece of training data:
[0065] Specifically, the similarity component between the training data and the corresponding literal translation corpus can be calculated using the following formula:
[0066]
[0067] in, Let w be the similarity component between the i-th training data and the corresponding translated corpus, where n is the total number of word vectors in that training data, and w is the similarity component between the i-th training data and the corresponding translated corpus. ij Let v be the word vector of the j-th word in the training data. ij For the direct translation corpus data and word vectors w ij The corresponding word vector, w ij ·v ij Word vector w ij With word vector v ij The dot product, ||w ij ||and||v ij || These are the word vectors w ij and word vector v ij The length of the module.
[0068] The similarity component between the literal translation corpus and the corresponding back-translated corpus is calculated using the following formula:
[0069]
[0070] in, Let v be the similarity component between the literal translation corpus and the back-translation corpus corresponding to the i-th training data, n be the total number of word vectors in the literal translation corpus, and v be the similarity component between the literal translation corpus and the back-translation corpus. ij The word vector w in the literal translation corpus and the training data is... ij The corresponding word vector, u ij For back-translation of corpus data and word vector v ij The corresponding word vector, v ij ·u ij Word vector v ij With word vector u ij The dot product, ||v ij || and ||u ij|| These are the word vectors v ij and word vector u ij The length of the module.
[0071] The similarity component between the training data and the corresponding back-translated corpus data is calculated using the following formula:
[0072]
[0073] in, Let w be the similarity component between the i-th training data and the corresponding back-translated corpus data, n be the total number of word vectors in that training data, and w be the similarity component between the i-th training data and the corresponding back-translated corpus data. ij Let u be the word vector of the j-th word in the training data. ij For back-translation corpus data and word vector w ij The corresponding word vector, w ij ·u ij Word vector w ij With word vector u ij The dot product, ||w ij || and ||u ij || These are the word vectors w ij and word vector u ij The length of the module.
[0074] Calculate the similarity components corresponding to the training data. Similarity components and similarity components Then, it can be based on similarity components Similarity components and similarity components Calculate the similarity value corresponding to the training data.
[0075] For example, the similarity value corresponding to the training data can be calculated using the following formula:
[0076]
[0077] Among them, S i Let be the similarity value corresponding to the i-th training data, where α, β, and γ are all weight parameters.
[0078] Furthermore, the training data and the corresponding back-translated corpus data can be output to obtain human evaluation. The human evaluation reveals whether the meaning expressed by the training data and the corresponding back-translated corpus data is consistent. It can be based on two dimensions: consistency and inconsistency (this method is adopted in this embodiment), or it can be further subdivided into more dimensions to assign different scores, such as completely consistent, basically consistent, partially inconsistent, completely inconsistent, etc. Alternatively, it can be scored directly based on the degree of consistency between the meaning expressed by the training data and the corresponding back-translated corpus data.
[0079] This approach to calculating similarity scores differs from the traditional source language corpus-translation corpus similarity measurement model. It introduces a similarity measurement model encompassing source language corpus (i.e., training data), translation corpus, and back-translation corpus data. A specific similarity measurement scheme is designed, and the weight allocation for different similarity components is considered, thus effectively taking into account the various similarity components. ( (Similarity components between the i-th training data and the corresponding translated corpus data) ( (These are the similarity components between the literal translation corpus and the corresponding back-translation corpus data corresponding to the i-th training data) and similarity components. ( The similarity components between the i-th training data and the corresponding back-translated corpus data are analyzed to determine the relationship between them and the influence of the component size on the weights, thereby more accurately determining the similarity value. This information is then used to update the word embedding space mapping matrix, improve the translation quality of the auxiliary translation model, and ultimately improve the quality of the pseudo-parallel corpus data.
[0080] Then, the word embedding space mapping matrix can be updated based on similarity values and human evaluation.
[0081] For example, the update range and direction of the word embedding space mapping matrix can be determined based on manual evaluation. The update range reveals which values exist in the updated word embedding space mapping matrix (related to the parameters in the word vector embedding space of the training data), and the update direction determines whether to increase or decrease the values within the update range. Then, based on the similarity value, the element update amount within the update range of the word embedding space mapping matrix is determined, and based on the element update amount, gradient descent is used to update the values within the update range along the update direction. This allows for the updating of the word embedding space mapping matrix, thereby optimizing the auxiliary translation model.
[0082] In this embodiment, the parallel corpus validation set and the source language corpus validation set can be combined into a corpus validation set. After the training achieves its target (e.g., the metric reaches a certain value, or the number of iterations reaches a certain threshold), the corpus validation set can be used to validate the auxiliary translation model. If the target is met, the training of the auxiliary translation model is complete. If the target is not met, the auxiliary translation model is further optimized using the corpus training set and the corpus training set, ultimately obtaining the trained auxiliary translation model.
[0083] After obtaining the trained auxiliary translation model, the source language monolingual corpus and the corresponding target language translation corpus can be determined based on the auxiliary translation model, and a pseudo-parallel corpus can be constructed.
[0084] For example, the remaining source language corpus data in the monolingual corpus dataset of the source language can be input into the auxiliary translation model one by one. The auxiliary translation model translates the input source language corpus data and outputs the corresponding target language translation corpus data. Then, each source language corpus data and the corresponding target language translation corpus data are used as a set of pseudo-parallel corpus data. This is combined with the original parallel corpus and the training data (and the corresponding direct translation corpus data) during the training process to construct a large-scale pseudo-parallel corpus.
[0085] After constructing a pseudo-parallel corpus of the source language and the target language, step S30 can be run.
[0086] Step S30: Use a pseudo-parallel corpus to adjust and train the primary translation model to obtain a trained cross-language translation model.
[0087] In this embodiment, a pseudo-parallel corpus can be used to train and adjust the initial translation model (the specific training process will not be described here), thereby obtaining a trained cross-language translation model. The cross-language translation model obtained in this way has stronger generalization and domain adaptability, and thanks to the idea of transfer learning, the translation quality can also be improved.
[0088] In summary, this application provides a cross-language translation model training method based on transfer learning. It pre-trains a machine translation model using parallel corpora of the source language and a pivot language (representing a selected high-resource language, which should be as close as possible to the target language in grammatical structure, e.g., English and Afrikaans; with Chinese as the source language, English is a high-resource language, while Afrikaans is a low-resource language). This results in a pre-trained primary translation model. Then, based on the parallel corpora of the source and target languages, a pseudo-parallel corpus of the source and target languages is constructed. This pseudo-parallel corpus is then used to fine-tune and train the primary translation model, resulting in a fully trained cross-language translation model. This approach utilizes the concept of transfer learning, a powerful machine learning method. Its core idea is to transfer knowledge or models learned on one task (source task) to another related task (target task) to improve the performance of the new task. This method fully utilizes existing data and model resources, reduces the target task's dependence on large amounts of new data, accelerates model training, and improves the model's generalization ability. Considering that even low-resource languages in cross-language translation tasks often have high-resource languages with similar grammatical structures, applying transfer learning to machine translation tasks of low-resource languages can yield good results. While transfer learning can reduce reliance on large amounts of new data, in practice, more data still helps improve translation accuracy. More data provides richer linguistic features and contextual information, enabling the model to better understand the source language and generate more accurate target language. When data in the target domain is very limited, using pseudo-parallel corpora can help the model better adapt to the language style and vocabulary usage of the target domain. This approach utilizes existing small-scale parallel corpora between the source and target languages to augment the data and construct pseudo-parallel corpora. This augmentation differs from simple data augmentation; it takes into account word meaning and grammatical structure, thereby expanding the available data in both the source and target languages and further improving translation quality.
[0089] By acquiring monolingual corpora of the source language and monolingual corpora of the target language, and using unsupervised learning algorithms (such as FastText, Word2Vec, etc.) to perform unsupervised word embedding pre-training on the source language monolingual corpora, the vector representation of each word is learned, and the semantic relationships between words are captured, resulting in word vectors for each source language word, forming a source language word vector set. Similarly, using unsupervised learning algorithms (such as FastText, Word2Vec, etc., but consistent with the unsupervised learning algorithm used for the source language monolingual corpora) to perform unsupervised word embedding pre-training on the target language monolingual corpora, the vector representation of each word is learned, and the semantic relationships between words are captured, resulting in word vectors for each target language word, forming a target language word vector set. Based on parallel corpora of the source and target languages, word vector sets of the source and target languages, and monolingual corpora of the source and target languages, an auxiliary translation model is trained. Based on this auxiliary translation model, the source language monolingual corpora and their corresponding target language translation corpora are determined, thus constructing a pseudo-parallel corpus. An auxiliary translation model is introduced to generate pseudo-translation corpora during the construction of the pseudo-parallel corpus. The purpose of this auxiliary translation model is to translate the source language monolingual corpora into the target language, thereby generating pseudo-parallel corpus pairs. The auxiliary translation model (used to generate pseudo-parallel corpus data) differs from the primary translation model (which serves as the starting point for transfer learning). The auxiliary translation model used to construct the pseudo-parallel corpus is specifically trained for this task. Of course, if an existing model is available, it can be used directly, for example, in scenarios where the source language is extended to the target language, such as in more refined, sub-domain translation tasks. The reason for using a pseudo-parallel corpus to adjust the pre-trained primary translation model is that it provides a solution for expanding training data under limited resources. By introducing pseudo-parallel corpora, the generalization ability of the model can be increased, enabling it to better handle unseen data. At the same time, the adjustment of the primary translation model by the pseudo-parallel corpora can achieve domain adaptation, thereby improving the domain adaptability of the cross-language translation model.
[0090] By using parallel corpora of the source and target languages, semantically corresponding word pairs between the two languages are identified. Furthermore, a word embedding space mapping matrix is determined, mapping from the source language's word embedding space to the target language's word embedding space. Based on this mapping matrix, the source language's word vector set, the target language's word vector set, and both the source and target language monolingual corpora, an auxiliary translation model is initialized and trained, resulting in the trained model. During initialization and training, this approach allocates a portion of the parallel corpora of the source and target languages, as well as the source language's monolingual corpus, as training data. This approach addresses the issue of sample imbalance, improves training efficiency, and avoids various problems caused by sample imbalance during the training phase of the auxiliary translation model. When training the translation assistance model using training data, the training scheme was also adjusted. The translation assistance model was pre-trained using a parallel corpus training set, and the word embedding space mapping matrix was updated accordingly. Then, the source language corpus training set was added to the parallel corpus training set and merged into a corpus training set. The translation assistance model was then trained iteratively, and the word embedding space mapping matrix was updated accordingly. The parallel corpus validation set and the source language corpus validation set were then merged into a corpus validation set, and the translation assistance model was validated using the corpus validation set until the training objective was achieved (for example, if the validation did not meet the target, a second training was required). Furthermore, in the process of training and updating the word embedding space mapping matrix, not only is similarity calculated, but also human evaluation is introduced (human evaluation reveals whether the meaning expressed by the training data and the corresponding back-translated corpus data is consistent), thereby determining the update range and direction of the updated word embedding space mapping matrix. The similarity value is used to determine the number of elements updated within the update range of the updated word embedding space mapping matrix, and a gradient descent strategy is adopted to update the word embedding space mapping matrix in order to train an auxiliary translation model with high translation accuracy, which can be used to generate pseudo-parallel corpus data and construct a pseudo-parallel corpus.
[0091] In the similarity value calculation process, unlike the traditional similarity measurement model of source language corpus data - literal translation corpus data, this paper introduces a similarity measurement model of source language corpus data (i.e., training data), literal translation corpus data, and back-translation corpus data. A similarity measurement scheme is designed, and the allocation of weights for different similarity components is considered, thus taking into account the similarity components well. ( (Similarity components between the i-th training data and the corresponding translated corpus data) ( (These are the similarity components between the literal translation corpus and the corresponding back-translation corpus data corresponding to the i-th training data) and similarity components. ( The similarity components between the i-th training data and the corresponding back-translated corpus data are analyzed to determine the relationship between them and the influence of the component size on the weights, thereby more accurately determining the similarity value. This information is then used to update the word embedding space mapping matrix, improve the translation quality of the auxiliary translation model, and ultimately improve the quality of the pseudo-parallel corpus data.
[0092] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for training a cross-language translation model based on transfer learning, characterized in that, include: By using parallel corpora of the source language and the pivot language, the machine translation model is pre-trained to obtain a pre-trained primary translation model, where the pivot language represents the selected high-resource language; Based on parallel corpora of source and target languages, a pseudo-parallel corpus of source and target languages is constructed, where the source language represents the language to be translated and the target language represents the language to which the translation is desired. The initial translation model was trained using a pseudo-parallel corpus to obtain a fully trained cross-language translation model.
2. The cross-language translation model training method based on transfer learning according to claim 1, characterized in that, Based on parallel corpora of the source and target languages, a pseudo-parallel corpus of the source and target languages is constructed, including: Obtain monolingual corpus datasets of the source language and the target language; Based on the monolingual corpus dataset of the source language, an unsupervised learning algorithm is used to perform unsupervised word embedding pre-training on the monolingual corpus of the source language, learn the vector representation of each word, capture the semantic relationship between words, and obtain the word vector of each word in the source language, forming a word vector set of the source language. Based on the target language monolingual corpus dataset, an unsupervised learning algorithm is used to perform unsupervised word embedding pre-training on the target language monolingual corpus, learn the vector representation of each word, capture the semantic relationship between words, obtain the word vector of each target language word, and form the target language word vector set. A translation assistance model is trained based on parallel corpora of the source and target languages, word vector sets of the source language and the target language, monolingual corpus datasets of the source language and the target language. Based on the translation assistance model, source language monolingual corpora and corresponding target language translation corpora are identified, and a pseudo-parallel corpus is constructed.
3. The cross-language translation model training method based on transfer learning according to claim 2, characterized in that, Based on parallel corpora of the source and target languages, word vector sets of the source and target languages, monolingual corpora of the source and target languages, an auxiliary translation model is trained, including: Based on parallel corpora of the source and target languages, semantically corresponding word pairs between the source and target languages are identified. Based on the semantically corresponding word pairs between the source language and the target language, the word embedding space mapping matrix from the word embedding space of the source language to the word embedding space of the target language is determined; Based on the word embedding space mapping matrix, the source language word vector set, the target language word vector set, the source language monolingual corpus dataset, and the target language monolingual corpus dataset, the auxiliary translation model is initialized and trained to obtain the trained auxiliary translation model.
4. The cross-language translation model training method based on transfer learning according to claim 3, characterized in that, Based on the word embedding space mapping matrix, the source language word vector set, the target language word vector set, the source language monolingual corpus dataset, and the target language monolingual corpus dataset, an auxiliary translation model is initialized and trained to obtain the trained auxiliary translation model, including: The auxiliary translation model is initialized based on the word embedding space mapping matrix, the source language word vector set, and the target language word vector set. Based on parallel corpora of source and target languages, parallel corpus data of source and target languages are identified, and the parallel corpus data is divided into parallel corpus training set and parallel corpus validation set according to a set ratio. In addition, a portion of source language corpus data is identified from the monolingual corpus dataset of source language, and this portion of source language corpus data is divided into source language corpus training set and source language corpus validation set according to a set ratio. The auxiliary translation model is pre-trained using a parallel corpus training set, and the word embedding space mapping matrix is updated accordingly. The source language corpus training set is added to the parallel corpus training set to form a corpus training set, which is then used to iteratively train the auxiliary translation model and update the word embedding space mapping matrix. The parallel corpus validation set and the source language corpus validation set are combined into a corpus validation set, and the auxiliary translation model is validated using the corpus validation set. If it meets the standard, the training of the auxiliary translation model is completed. If it does not meet the standard, the auxiliary translation model is trained a second time using the corpus training set and the corpus training set to obtain the trained auxiliary translation model.
5. The cross-language translation model training method based on transfer learning according to claim 4, characterized in that, The source language corpus training set is added to the parallel corpus training set to form a new corpus training set. This corpus is then used to iteratively train the auxiliary translation model, and the word embedding space mapping matrix is updated accordingly, including: The source language corpus training set is added to the parallel corpus training set and merged into a corpus training set; For each piece of training data in the training corpus, it is input into the auxiliary translation model to obtain the corresponding literal translation corpus data. Then, the literal translation corpus data is translated back into the source language to obtain the back-translated corpus data corresponding to the training data. Based on the training data, the literal translation corpus data and the corresponding back-translation corpus data, similarity is calculated to obtain the similarity value corresponding to the training data, and human evaluation is obtained. The word embedding space mapping matrix is updated based on similarity values and human evaluation.
6. The cross-language translation model training method based on transfer learning according to claim 5, characterized in that, Based on the training data, the literal translation corpus, and the corresponding back-translation corpus, similarity is calculated to obtain the similarity value corresponding to the training data, including: Determine the word vector for each word in the training data, determine the word vector for each word in the literal translation corpus, and determine the word vector for each word in the back-translation corpus. The correspondence between word vectors in the training data, word vectors in the literal translation corpus, and word vectors in the back-translation corpus was determined. Based on the word vectors of the corresponding training data, the word vectors of the direct translation corpus, and the word vectors of the back translation corpus, the similarity value of the corresponding training data is calculated.
7. The cross-language translation model training method based on transfer learning according to claim 6, characterized in that, Based on the word vectors of the corresponding training data, the word vectors of the literal translation corpus, and the word vectors of the back-translation corpus, the similarity value corresponding to the training data is calculated, including: The similarity component between the training data and the corresponding literal translation corpus is calculated using the following formula: in, Let w be the similarity component between the i-th training data and the corresponding translated corpus, where n is the total number of word vectors in that training data, and w is the similarity component between the i-th training data and the corresponding translated corpus. ij Let v be the word vector of the j-th word in the training data. ij For the direct translation corpus data and word vectors w ij The corresponding word vector, w ij ·v ij Word vector w ij With word vector v ij The dot product, ||w ij ||and||v ij || These are the word vectors w ij and word vector v ij The modulus length; The similarity component between the literal translation corpus and the corresponding back-translated corpus is calculated using the following formula: in, Let v be the similarity component between the literal translation corpus and the back-translation corpus corresponding to the i-th training data, n be the total number of word vectors in the literal translation corpus, and v be the similarity component between the literal translation corpus and the back-translation corpus. ij The word vector w in the literal translation corpus and the training data is... ij The corresponding word vector, u ij For back-translation of corpus data and word vector v ij The corresponding word vector, v ij ·u ij Word vector v ij With word vector u ij The dot product, ||v ij || and ||u ij || These are the word vectors v ij and word vector u ij The modulus length; The similarity component between the training data and the corresponding back-translated corpus is calculated using the following formula: in, Let w be the similarity component between the i-th training data and the corresponding back-translated corpus data, n be the total number of word vectors in that training data, and w be the similarity component between the i-th training data and the corresponding back-translated corpus data. ij For the j-th word vector in the training data, u ij For back-translation corpus data and word vector w ij The corresponding word vector, w ij ·u ij Word vector w ij With word vector u ij The dot product, ||w ij || and ||u ij || These are the word vectors w ij and word vector u ij The modulus length; Based on similarity components Similarity components and similarity components Calculate the similarity value corresponding to the training data.
8. The cross-language translation model training method based on transfer learning according to claim 7, characterized in that, Based on similarity components Similarity components and similarity components Calculate the similarity value corresponding to the training data, including: The similarity value corresponding to the training data is calculated using the following formula: Among them, S i Let be the similarity value corresponding to the i-th training data, where α, β, and γ are all weight parameters.
9. The cross-language translation model training method based on transfer learning according to claim 7, characterized in that, Human evaluation reveals whether the meaning expressed by the training data and the corresponding back-translated corpus data is consistent. Based on similarity values and human evaluation, the word embedding space mapping matrix is updated, including: Based on manual evaluation, the update range and update direction of the word embedding space mapping matrix are determined. The update range is used to reveal the value of the word embedding space mapping matrix in this update, and the update direction is used to determine whether to increase or decrease the value within the update range. Based on the similarity value, determine the number of elements to be updated within the update range of the word embedding space mapping matrix for this update; Based on the element update amount, the gradient descent method is used to update the values within the update range along the update direction.
10. The cross-language translation model training method based on transfer learning according to claim 2, characterized in that, Based on the translation-assisted model, source language monolingual corpora and corresponding target language translation corpora are determined, and a pseudo-parallel corpus is constructed, including: The remaining source language corpus data from the monolingual corpus dataset of the source language are input into the auxiliary translation model one by one; The assisted translation model translates the input source language corpus data and outputs the corresponding target language translation corpus data; Each source language corpus data and its corresponding target language translation corpus data are treated as a set of pseudo-parallel corpus data, and a pseudo-parallel corpus is constructed based on this.