Probability-driven metagraph prompting method for low-resource unsupervised machine translation

Abstract

The invention discloses a low-resource unsupervised machine translation-oriented probability-driven metagraph prompting method, which relates to the technical field of machine translation, and is characterized in that an unsupervised neural machine translation model is constructed and trained, and initial translation and pseudo-parallel statements are generated; constructing a language-specific meta-graph, and generating a diversified translation path for promoting translation of a low-resource language by using a high-resource-assisted language; sampling a plurality of translation paths from the metagraph, executing a generation operation and an aggregation operation, and generating an optimized translation by using a large language model; and performing back propagation on the score obtained in the aggregation operation, updating the probability of the auxiliary language in the meta-graph, and optimizing the translation process of the next round of iteration until training convergence. According to the method, pseudo-parallel sentences and cross-language information of auxiliary languages are dynamically organized, the unsupervised neural machine translation technology is improved by utilizing a meta-graph prompting method aiming at the translation quality of low-resource languages, language noise can be effectively reduced, and translation accuracy and smoothness are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine translation, and more particularly, to a probability-driven meta-graph prompting method for low-resource unsupervised machine translation. Background Art

[0002] In recent years, large language models with ultra-large-scale parameters, such as GPT-3.5, GPT-4, and LLaMA, have made significant breakthroughs in the field of natural language processing. They use contextual learning mechanisms, prompt learning mechanisms, and directional propagation mechanisms to learn rich language, knowledge, and reasoning capabilities through pre-training on large-scale datasets. The contextual learning mechanism refers to the ability of the model to dynamically adjust its behavior based on the input context provided (such as a few examples or prompts) to complete specific tasks without explicit training. This learning mechanism allows the model to quickly adapt to new tasks or new fields using a small number of examples, significantly improving the flexibility and applicability of the model; the prompt learning mechanism involves embedding task instructions or prompts into the input of the model to guide the output direction of the model. Through carefully designed prompts, the model can better understand the task requirements and contextual information, thereby performing better on specific tasks. This mechanism is particularly effective for zero-shot or few-shot learning scenarios, and can significantly improve the performance of large language models on unseen tasks; the backpropagation mechanism is an effective parameter optimization strategy in the field of deep learning, used to train neural networks. Specifically, by calculating the gradient of the loss function with respect to the model parameters, the parameters are updated in the opposite direction of the gradient to minimize the loss function. This mechanism is the core training algorithm of many deep learning models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), and variational autoencoders (VAE); these models can understand and generate human language and are widely used in multiple tasks such as text generation, question-answering systems, and machine translation.

[0003] The rapid development of large language models has brought about more application scenarios, especially in the field of machine translation of low-resource languages. By further optimizing and adjusting these large language models, they can be more suitable for specific application scenarios. However, there are currently a wide range of problems in machine translation of low-resource languages. Even though the existing unsupervised translation methods provide solutions for low-resource languages ​​without a large amount of parallel data, they still face the following major shortcomings:

[0004] 1. The back-translation method relies on the existing "target-source" machine translation model to generate parallel data for training, but this method is limited by the incorrect translations in the synthetic data;

[0005] 2. The machine translation method based on transfer learning is trained on a high-resource auxiliary language and transferred to a low-resource language for reasoning. However, the language deviation between the training language (auxiliary language) and the test language leads to poor translation accuracy.

[0006] 3. The pivot-based translation method first translates the source language into the pivot (auxiliary) language and then translates it back into the source language. The multi-hop translation process introduces accumulated translation errors between different languages.

[0007] The above shortcomings make existing unsupervised translation methods face multiple language noise problems when dealing with low-resource languages, including translation errors of synthetic data in back-translation methods, deviations between training and testing languages ​​in transfer learning methods, and errors introduced by the multi-hop translation process in pivot-based translation methods.

[0008] Therefore, the present invention aims to provide a probability-driven meta-graph prompting method for low-resource unsupervised machine translation to solve the above problems. Summary of the invention

[0009] The purpose of the present invention is to provide a probability-driven meta-graph prompting method for low-resource unsupervised machine translation. The present invention improves the unsupervised neural machine translation technology through the meta-graph prompting method, especially the translation quality of low-resource languages. It utilizes pseudo-parallel sentences and cross-language information of dynamically organized auxiliary languages ​​to effectively reduce language noise and improve the accuracy and fluency of translation, thereby solving the main technical shortcomings of the prior art when processing low-resource language translation.

[0010] The above technical objectives of the present invention are achieved through the following technical solutions: a probability-driven meta-graph prompting method for low-resource unsupervised machine translation, comprising the following steps:

[0011] S1. Build and train an unsupervised neural machine translation model to generate initial translations from the source language to the target language, as well as pseudo-parallel sentences from the source language to the auxiliary language pair;

[0012] S2, using pseudo-parallel sentences to build language-specific metagraphs and generate diversified translation paths that use high-resource auxiliary languages ​​to promote low-resource language translation;

[0013] S3, by sampling multiple translation paths from the meta-graph, performing generation and aggregation operations, and then using the large language model to generate optimized translations for low-resource languages;

[0014] S4. Through the graph optimization strategy based on probability feedback, the scores obtained in the aggregation operation are back-propagated to update the probabilities of the auxiliary languages ​​in the meta-graph, and the translation process of the next iteration is optimized until the training converges. The translation path sampled in the last iteration is used for testing to complete the probability-driven meta-graph prompting method for low-resource unsupervised machine translation.

[0015] The present invention is further configured as follows: the process of constructing and training an unsupervised neural machine translation model in step S1 is as follows: constructing a Transformer-based neural machine translation model, using the weights of the multilingual pre-trained model XLM-R as initialization parameters, and then applying the first-stage training method and the second-stage training method to six auxiliary language data sets to train the model.

[0016] The present invention is further configured as follows: the process of the first-stage training method is: training the decoder by training the auxiliary language to English pairs; the process of the second-stage training method is: jointly optimizing all parameters of the encoder and decoder, updating the parameters through synchronous training, and transferring the translation ability learned by the model from the auxiliary language to the low-resource language under test.

[0017] The present invention is further configured as follows: the metagraph in step S2 is composed of vertices, edges and weights; the vertices represent the languages ​​in the translation process, the edges represent the paths of conditional transfer from the current language to the next language, and the weights represent the conditional probability of transfer from the current language to the next language. In the metagraph, the translation path starts from the source language, passes through multiple auxiliary languages, and finally reaches the target language.

[0018] The present invention is further configured as follows: the method for constructing a language-specific meta-graph in step S2 comprises the following steps:

[0019] S201, creating a source vertex, a target vertex, and an auxiliary vertex representing a source language, a target language, and an auxiliary language respectively;

[0020] S202, connecting the source vertex with a plurality of different auxiliary vertices representing auxiliary languages, to complete the establishment of edges in the metagraph;

[0021] S203, further extending each auxiliary vertex to be connected to the target vertex or all other auxiliary vertices that are not connected in the previous path, so as to complete the path growth in the meta-graph;

[0022] S204, determine whether the path reaches the target vertex, if so, the path is considered complete and step S305 is executed, otherwise, jump to step S203;

[0023] S205. Assign a weight to each edge in the meta-graph, thereby completing the construction of a language-specific meta-graph.

[0024] The present invention is further configured as follows: the method for calculating the probability of the auxiliary language in step S4 is: first, multiple pseudo-parallel sentences of the generated source language-auxiliary language pairs are encoded to obtain vector pairs; secondly, the average value of the cosine similarity of all vector pairs is calculated to obtain the similarity used to measure the languages; finally, the similarity is scaled to obtain the probability of the auxiliary language.

[0025] The present invention is further configured as follows: the process of generating the operation in step S3 is: through the context learning method, for each auxiliary vertex of the auxiliary language on the sampling path, 4 examples and 1 input are used to construct prompt words, and a number of translation results corresponding to the auxiliary vertices are generated; wherein each example includes a source sentence, a pseudo-parallel auxiliary sentence at the vertex, a translation output and an optimized translation as a pseudo-reference, the input includes an input source sentence and other parts of the example except the optimized translation, and the large language model is prompted to generate an optimized translation for the input.

[0026] The present invention is further configured as follows: the process of the aggregation operation in step S4 is: through the context learning method, the pseudo-parallel sentences of all auxiliary languages ​​on the sampling path are aggregated as prompt words; wherein, the aggregation operation of a sampling path is performed once, and only one translation is generated accordingly.

[0027] The present invention is further configured as follows: the method for updating the probability of the auxiliary language in the meta-graph in step S4 comprises the following steps:

[0028] S401, the auxiliary languages ​​in the preset translation path have equal contributions to the output, the scores of the aggregation operation are evenly distributed to the contributions of each language, the scores of the generation operation are used to obtain the specific contributions of each auxiliary language, the translation scores without considering each language are calculated, the contributions of each language are determined, and the determined contributions are used to calculate each language to generate the specific contributions of each language;

[0029] S402, using the central symmetric Swish function to normalize the specific contribution of the language to the interval of [0, 1], and obtain the standardized reward corresponding to the specific contribution;

[0030] S403: Use the learning rate and the original probability of the auxiliary language to back-propagate the corresponding standardized reward, and then calculate and update the new probability of the current auxiliary language for the next iteration.

[0031] In summary, the present invention has the following beneficial effects:

[0032] 1. The present invention combines an unsupervised neural machine translation model with a large language model and constructs prompt words in a dynamic graph manner, thereby providing an innovative solution for low-resource language translation. The unsupervised translation model is used to generate pseudo-parallel sentences, so that the invention can effectively use high-resource auxiliary languages ​​to enhance the translation quality of low-resource languages ​​in the absence of a large amount of parallel corpus. In addition, the present invention not only provides a variety of translation path options by constructing a language-specific metagraph, but also effectively integrates auxiliary language information through "generation" and "aggregation" operations, further improving the accuracy and fluency of translation;

[0033] 2. The present invention utilizes graph optimization technology based on probability feedback and dynamically updates the probability of auxiliary languages, so that the model can continuously optimize itself according to the actual translation effect, thereby improving the translation quality in a continuous iterative process. This enables the present invention to demonstrate significant advantages in improving the translation quality of low-resource languages, and at the same time has important application value and prospects for low-resource language translation and multilingual information processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the probability-driven meta-graph prompting method for low-resource unsupervised machine translation in Example 1 of the present invention. DETAILED DESCRIPTION

[0035] The following is combined with Figure 1 The present invention is described in further detail.

[0036] Embodiment: A probability-driven meta-graph prompting method for low-resource unsupervised machine translation comprises the following steps:

[0037] S1. Build an unsupervised neural machine translation model and train it to generate an initial translation from the source language to the target language, as well as pseudo-parallel sentences from the source language to the auxiliary language pair; wherein the auxiliary language is an additional language used to assist in activating the generation capability of the large language model. In this embodiment, a Transformer-based neural machine translation model is adopted, and the weights of the multilingual pre-trained model XLM-R are used as initialization parameters. Then, a two-stage training method is applied to the six auxiliary language datasets for model training.

[0038] In the first stage, in order to maintain the cross-language transfer capability of the encoder, in this embodiment, the decoder is only trained for pairs from the auxiliary language to English; in the second stage, in order to further learn from the training data, all parameters of the encoder and decoder are jointly optimized and synchronized for parameter update. This processing method can transfer the translation capability learned from the auxiliary language to the low-resource language under test, and then use the unsupervised translation model to generate translations of the source text in the low-resource language, and further generate pseudo-parallel sentences from the source language to the auxiliary language pair for constructing metagraphs and generating prompt words.

[0039] S2. Use pseudo-parallel statements to build a language-specific metagraph. The metagraph structure in this embodiment consists of vertices (representing the language in the translation process), edges (representing the path of conditional transfer from the current language to the next language) and weights (representing the conditional probability of transfer from the current language to the next language). The translation path starts from the source language, passes through multiple auxiliary languages, and finally reaches the target language. The weight of each edge reflects the conditional transfer probability of all previous vertices in the path, generating a diversified translation path that uses high-resource auxiliary languages ​​to promote low-resource language translation.

[0040] In this embodiment, constructing a language-specific metagraph includes the following five steps:

[0041] 1. Vertex initialization: Create three vertices, namely the source vertex, target vertex and auxiliary vertex, representing the source language, target language and auxiliary language respectively;

[0042] 2. Edge creation: connect the source vertex to m different auxiliary vertices representing auxiliary languages; each auxiliary vertex has a corresponding probability, and the edge represents a directed connection from the existing vertex to the newly connected vertex;

[0043] 3. Path growth: further extend each auxiliary vertex to connect it to the target vertex or all other auxiliary vertices that are not connected in the previous path;

[0044] 4. Path completion: If the path reaches the target vertex, the path is considered complete, otherwise it continues to expand according to the path growth steps;

[0045] 5. Edge weight assignment: Assign a weight to each edge, which is the geometric mean of the probabilities of all previous vertices in the path.

[0046] In the constructed meta-graph, the language similarity between the auxiliary language and the source language is taken into account in this embodiment to calculate the probability of the auxiliary language. This probability is intended to guide the organizational structure of the auxiliary language when constructing the prompt words of the large language model. The specific operations are as follows:

[0047] First, this embodiment encodes the n pseudo-parallel sentences of the source language-auxiliary language pair generated in the first step to obtain vector pairs; secondly, in order to measure the similarity between languages, this embodiment calculates the average value of the cosine similarity of all vector pairs; finally, this embodiment scales the similarity into the probability of the auxiliary language.

[0048] Through the above operations, this embodiment constructs a language-specific meta-graph that can reflect the relationship between the source language, the target language and the auxiliary language, which is used to generate multiple translation paths and effectively guide the large language model in the translation process.

[0049] S3. By sampling multiple translation paths from the metagraph, performing generation operations and aggregation operations, and then using the large language model to generate optimized translations for low-resource languages; in this embodiment, specifically, multiple paths are independently sampled from the metagraph of a specific language. In the generation operation, this embodiment uses the auxiliary language of a single vertex on the path to construct a prompt word. In the aggregation operation, this embodiment constructs a prompt word by combining the auxiliary languages ​​of all vertices on the path, thereby stimulating the large language model to generate multiple optimized low-resource translations.

[0050] In order to explore different prompting methods for optimizing low-resource translation, this embodiment samples multiple translation paths from the metagraph according to the probability of the path. The probability of the path is the geometric mean of the probabilities of all auxiliary vertices in the path. These sampled paths provide different auxiliary languages ​​and their combinations for constructing prompt words in generation and aggregation operations.

[0051] In the generation operation, in order to fully utilize the auxiliary language and explore the capabilities of the large language model, this embodiment performs a generation operation on each vertex on the sampling path, such as Figure 1 As shown, according to the context learning method, 4 examples and 1 input are used to construct prompt words, wherein each example includes 4 parts: a source sentence, a pseudo-parallel auxiliary sentence at a vertex, a translation output, and an optimized translation (Google Translate) as a pseudo-reference. The input includes an input source sentence and other parts of the example except for the optimized translation. The large language model is prompted to generate an optimized translation for the input. In this embodiment, each auxiliary language vertex on a path performs a generation operation once to generate m translation results.

[0052] In the aggregation operation, in order to utilize multiple auxiliary languages ​​and fuse useful information to prompt the large language model, the present embodiment performs an aggregation operation, in which the pseudo-parallel sentences of all auxiliary languages ​​on the entire sampling path are aggregated into prompt words. Similar to the above-mentioned generation operation, the aggregation operation also adopts the context learning method, but the difference between aggregation and generation is that the aggregation operation aggregates the pseudo-parallel sentences of all auxiliary languages ​​on the path as prompt words, while the generation operation uses a single auxiliary language sentence at each vertex; the input in the aggregation operation includes the optimized translation obtained from the generation operation instead of the original translation; in addition, the aggregation operation of a path is performed only once to generate a translation.

[0053] In the unsupervised evaluation without a real reference, this embodiment uses the translations in the m generation operations as pseudo references in turn, and the average score of the remaining m-1 translations obtained in this way is used as the unsupervised evaluation score of the translation selected as the pseudo reference. In the aggregation operation, this embodiment uses its single translation as the pseudo reference, and calculates the average score of the m translations generated by the generation operation as the unsupervised evaluation score. During the training process, the evaluation score of the aggregation output will be back-propagated to update the probability of the auxiliary language involved. During the test, this embodiment selects the one with the highest score from the m generation operation outputs and 1 aggregation operation output as the final output.

[0054] S4. Through the graph optimization strategy based on probability feedback, the scores obtained in the aggregation operation are back-propagated to update the probabilities of the auxiliary languages ​​in the meta-graph, thereby optimizing the sampling probabilities of different translation paths in the meta-graph. This process includes the following three steps: (1) calculating the contribution of each auxiliary language to the translation; (2) normalizing these contributions into rewards; (3) back-propagating the rewards to update the probabilities of the auxiliary languages ​​in the meta-graph.

[0055] Among them, since all auxiliary languages ​​in the sampled translation path contribute to the output of the aggregation operation, this embodiment first assumes that the contribution of the m auxiliary languages ​​in the translation path to the output is equal, and the score E of the aggregation operation is evenly distributed to the contribution of each language; then, the score e of the generation operation is used to generate the output of the sampled translation path. i Reflect the specific contribution of each auxiliary language i by calculating the translation score Ee without considering a certain language i , to determine the contribution of the language; by performing the following calculation on all languages, this embodiment can obtain the specific contribution d of each language i i :

[0056]

[0057] The solution is:

[0058]

[0059] In order to contribute i Convert to reward i In this embodiment, the centrosymmetric Swish function (cs-Swish) is used to convert d i Normalize to the interval [0,1] to ensure that the updated probability is in the range [0,1], and the normalized reward r i According to the contribution i Calculated; the calculation process is as follows:

[0060]

[0061] Bring in:

[0062]

[0063] For a training sample and its sampled translation path, this embodiment obtains m corresponding rewards r for m auxiliary languages ​​in the path. i After that, use the learning rate (lr) and the original probability of the auxiliary language To back propagate r i , update the new probability of the i-th auxiliary language After one training iteration, the probabilities of the auxiliary languages ​​involved in the meta-graph are updated for the next iteration. When the training converges, this embodiment uses the translation paths sampled in the last iteration to complete subsequent testing.

[0064] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A probability-driven meta-graph hinting method for low-resource unsupervised machine translation, characterized by: The following steps are involved: S1. Build and train an unsupervised neural machine translation model to generate initial translations from the source language to the target language, as well as pseudo-parallel sentences from the source language to the auxiliary language pair; S2, using pseudo-parallel sentences to build language-specific metagraphs and generate diversified translation paths that use high-resource auxiliary languages ​​to promote low-resource language translation; S3, by sampling multiple translation paths from the meta-graph, performing generation and aggregation operations, and then using the large language model to generate optimized translations for low-resource languages; S4. Through the graph optimization strategy based on probability feedback, the scores obtained in the aggregation operation are back-propagated to update the probabilities of the auxiliary languages ​​in the meta-graph, and the translation process of the next iteration is optimized until the training converges. The translation path sampled in the last iteration is used for testing to complete the probability-driven meta-graph prompting method for low-resource unsupervised machine translation.

2. The probability-driven meta-graph prompting method for low-resource unsupervised machine translation according to claim 1, characterized in that: The process of constructing and training the unsupervised neural machine translation model in step S1 is as follows: constructing a Transformer-based neural machine translation model, using the weights of the multilingual pre-trained model XLM-R as initialization parameters, and then applying the first-stage training method and the second-stage training method to the six auxiliary language datasets to train the model.

3. The probability-driven meta-graph prompting method for low-resource unsupervised machine translation according to claim 2, characterized in that: The process of the first stage training method is as follows: training the decoder by pairing auxiliary language to English; The process of the second-stage training method is: jointly optimize all parameters of the encoder and decoder, perform synchronous training to update the parameters, and transfer the translation ability learned by the model from the auxiliary language to the low-resource language under test.

4. The probability-driven meta-graph prompting method for low-resource unsupervised machine translation according to claim 1, characterized in that: The meta-graph in step S2 is composed of vertices, edges and weights; the vertices represent the languages ​​in the translation process, the edges represent the paths of conditional transfer from the current language to the next language, and the weights represent the conditional probability of transfer from the current language to the next language. In the meta-graph, the translation path starts from the source language, passes through multiple auxiliary languages, and finally reaches the target language.

5. The probability-driven meta-graph prompting method for low-resource unsupervised machine translation according to claim 1, characterized in that: The method for constructing a language-specific meta-graph in step S2 comprises the following steps: S201, creating a source vertex, a target vertex, and an auxiliary vertex representing a source language, a target language, and an auxiliary language respectively; S202, connecting the source vertex with a plurality of different auxiliary vertices representing auxiliary languages, to complete the establishment of edges in the metagraph; S203, further extending each auxiliary vertex to be connected to the target vertex or all other auxiliary vertices that are not connected in the previous path, so as to complete the path growth in the meta-graph; S204, determine whether the path reaches the target vertex, if so, the path is considered complete and step S305 is executed, otherwise, jump to step S203; S205. Assign a weight to each edge in the meta-graph, thereby completing the construction of a language-specific meta-graph.

6. The probability-driven meta-graph prompting method for low-resource unsupervised machine translation according to claim 1, characterized in that: The method for calculating the probability of the auxiliary language in step S4 is as follows: first, multiple pseudo-parallel sentences of the generated source language-auxiliary language pairs are encoded to obtain vector pairs; secondly, the average cosine similarity of all vector pairs is calculated to obtain the similarity used to measure the languages; finally, the similarity is scaled to obtain the probability of the auxiliary language.

7. The probability-driven meta-graph prompting method for low-resource unsupervised machine translation according to claim 1, characterized in that: The process of the generation operation in step S3 is: for each auxiliary vertex of the auxiliary language on the sampling path, four examples and one input are used to construct prompt words, and the translation results corresponding to the number of auxiliary vertices are generated; Each example includes a source sentence, a pseudo-parallel auxiliary sentence at the vertex, a translation output, and an optimized translation as a pseudo-reference. The input contains an input source sentence and the rest of the example except the optimized translation. The large language model is prompted to generate an optimized translation for the input.

8. The probability-driven meta-graph prompting method for low-resource unsupervised machine translation according to claim 1, characterized in that: The process of the aggregation operation in step S4 is: by using the context learning method, the pseudo-parallel sentences of all auxiliary languages ​​on the sampling path are aggregated as prompt words; wherein, the aggregation operation of a sampling path is performed once, and only one translation is generated accordingly.

9. The probability-driven meta-graph prompting method for low-resource unsupervised machine translation according to claim 1, characterized in that: The method for updating the probability of the auxiliary language in the meta-graph in step S4 comprises the following steps: S401, the auxiliary languages ​​in the preset translation path have equal contributions to the output, the scores of the aggregation operation are evenly distributed to the contributions of each language, the scores of the generation operation are used to obtain the specific contributions of each auxiliary language, the translation scores without considering each language are calculated, the contributions of each language are determined, and the determined contributions are used to calculate each language to generate the specific contributions of each language; S402, using the central symmetric Swish function to normalize the specific contribution of the language to the interval of [0, 1], and obtain the standardized reward corresponding to the specific contribution; S403: Use the learning rate and the original probability of the auxiliary language to back-propagate the corresponding standardized reward, and then calculate and update the new probability of the current auxiliary language for the next iteration.

Images