Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of large language model
By fine-tuning the large language model on the Mongolian and Chinese parallel corpus and building multi-dimensional tips, the problem of large language model performing poorly in Mongolian and Chinese translation is solved, and the translation quality and consistency are improved.
Patent Information
- Application Number
- CN202510426708.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-07
- Publication Date
- 2025-07-25
AI Technical Summary
The large language model performed poorly in Mongolian and Chinese translation tasks and failed to fully utilize its semantic understanding ability, resulting in insufficient translation consistency and fluency.
The low-rank adaptation method supervises fine-tune pre-trained large language model on the Mongolian and Chinese parallel corpus, constructs word meaning chains and syntactic pattern prompts, and guides the model for translation in combination with clear instruction prompts.
It significantly improves the quality and consistency of Mongolian and Chinese translation, optimizes the low-resource Mongolian translation effect, reduces calculation overhead and avoids negative optimization problems.
Smart Images
Figure CN120373323A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of neural machine translation, and particularly relates to a Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of a large language model. Background Art
[0002] Machine translation is the process of using a computer to map from one language to another. Compared with traditional rule-based translation systems, neural machine translation based on deep learning has become the mainstream technology due to its significant advantages in translation quality and multilingual adaptation.
[0003] In recent years, with the rapid development of deep learning technology, large language models with huge parameter scales have emerged. They have strong generalization abilities and can capture the deep semantic and syntactic structures of languages. Therefore, they are widely used in many natural language processing fields such as sentiment analysis, text summarization generation, and machine translation. However, due to the lack of low-resource language data in the pre-training stage of large language models, their performance in Mongolian-Chinese translation tasks is still not ideal, and they fail to fully utilize the advantages of their semantic understanding abilities. Therefore, choosing an appropriate method to exploit the translation potential of large language models has become a key to improving the effect of Mongolian-Chinese neural machine translation. Summary of the Invention
[0004] In order to overcome the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide a Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of a large language model. By using the semantic chain of a high-resource language and the syntactic pattern embedding of the target sentence to prompt the template, it helps the model better solve the complex semantic features of Mongolian and improve translation consistency and fluency.
[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:
[0006] A Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of a large language model, comprising the following steps:
[0007] Step 1, supervise and fine-tune a pre-trained large language model M on a large-scale Mongolian-Chinese parallel corpus through the low-rank adaptation method.
[0008] Step 2, construct prompts from the semantic dimension. For each sentence x in the Mongolian corpus i Extract several keywords; translate each keyword into corresponding words in two higher-resource auxiliary languages and merge them into a semantic chain prompt; the higher-resource languages refer to languages with richer corpora than Mongolian, such as mainstream languages like English, French, Japanese, etc.
[0009] Step 3: For the Mongolian sentence x to be translated currently, use the semantic embedding model to map x and the entire Chinese target monolingual corpus Dy into the same semantic space; calculate the similarity scores between the Mongolian source sentence x and each sentence in the Chinese target monolingual corpus Dy, and obtain the K most similar sentences as examples.
[0010] Step 4: Design instruction prompts to guide the pre-trained large language model M to parse the syntactic patterns of the K sentence examples respectively, and combine them to form syntactic pattern prompts.
[0011] Step 5: Design explicit instruction type prompts to simply and clearly describe the task objective and directly guide the pre-trained large language model M to perform translation operations.
[0012] Step 6: Concatenate the corresponding semantic chain prompts, syntactic pattern prompts, and explicit instruction type prompts for each Mongolian sentence x i and input them into the pre-trained large language model M to obtain high-quality Mongolian-Chinese translation results.
[0013] In one embodiment, in the above Step 1, a pre-trained large language model based on the Transformer architecture is adopted, and the self-attention mechanism is used to dynamically capture the semantic associations between each word and other words in the input sequence. The formula is:
[0014]
[0015] where Q, K, and V respectively represent the query matrix, key matrix, and value matrix, which are the embedding vectors from the input sequence; d k is the dimension of the key vector, which is used for normalization to stabilize the gradient;
[0016] For the low-rank adaptation method, knowledge of the Mongolian-Chinese translation task is captured by adding two low-rank matrices, and finally, the product of the two low-rank matrices A and B is added to the original weight matrix of the pre-trained large language model to complete the fine-tuning of the model.
[0017] In one embodiment, in the above Step 2, the extraction of sentence keywords is implemented by the term frequency-inverse document frequency (TF-IDF) method. The formula is:
[0018]
[0019] where f(t, d) represents the frequency of occurrence of word t in sentence d, N d represents the total number of words in sentence d, N is the total number of sentences in the corpus, and df(t) is the total number of sentences containing word t;
[0020] The number of keywords obtained depends on the sentence length and is extracted proportionally; a weight threshold is set to filter out noise such as the most frequently occurring function words and numbers. Each keyword is translated into the target Chinese vocabulary and the corresponding vocabulary of two higher-resource auxiliary languages through a multilingual knowledge base tool, and merged into a semantic chain prompt.
[0021] The semantic chain prompt is in the following format:
[0022] "{The following gives the keyword and its translations in different languages:}
[0023] ‘<Mongolian word X source >’ corresponds to the Chinese interpretation of ‘<Chinese corresponding word X target >’,
[0024] and is equivalently interpreted as ‘<Auxiliary language 1 corresponding word X1>’.
[0025] and is equivalently interpreted as ‘<Auxiliary language 2 corresponding word X2>’.
[0026] In one embodiment, in step 3, the cosine similarity formula is used to calculate the similarity score between the Mongolian source sentence vector and all Chinese target sentence vectors. The formula is as follows:
[0027]
[0028] where v mo and v zh are high-dimensional semantic vectors of Mongolian and Chinese generated by an embedding model; |||| represents the norm of the vector.
[0029] In one embodiment, in step 4, the instruction prompt content is a natural language instruction. The large language model M predicts the part of speech of each word in the sentence according to the instruction, combined with the context information and its learned language knowledge, and the result is output in the form of a standard part of speech tag (such as noun, verb, adjective, etc.). The instruction content is as follows:
[0030] "For a given set of Mongolian-Chinese parallel example sentences, parse K Chinese examples into syntactic patterns."
[0031] The result is output in the form of a standard part of speech tag. The syntactic analysis format of each Chinese sentence is as follows:
[0032] "{Syntactic pattern:} Subject: <Subject>, Verb: <Verb>, Object: <Object>, Time / Adverbial phrase: <Time / Adverbial phrase>, Subordinate clause subject: <Subordinate clause subject>,
[0033] Subordinate clause verb: <Subordinate clause verb>, Subordinate clause object: <Subordinate clause object>"
[0034] Among them, <...> represents the corresponding vocabulary in the sentence. After further sorting, the syntactic pattern prompt format is as follows:
[0035] "Given K example sentences and their Chinese syntactic patterns:
[0036] {Example 1:} <Chinese sentence 1> {Syntactic pattern 1:} <Syntactic analysis 1>
[0037] ... {Example K:} <Chinese sentence K> {Syntactic pattern K:} <Syntactic analysis K>"
[0038] In one embodiment, the explicit instruction type prompt content in step 5 is a natural language instruction, which clearly guides the pre-trained large language model M to perform the correct task. The prompt content is as follows:
[0039] "Please translate the following Mongolian sentence into Chinese: {Mongolian:} <Mongolian sentence> {Chinese:}"
[0040] Among them, "{Chinese:}" is the flag for the pre-trained large language model M to start performing the translation.
[0041] In one embodiment, in step 6, for each sentence x to be translated i The text content of the semantic chain prompt, syntactic pattern prompt, and explicit instruction type prompt corresponding to it is concatenated and combined in sequence through python code to form the overall input content of the sentence. Subsequently, this integrated prompt is input into the pre-trained large language model M to guide the model to generate a more accurate translation output.
[0042] In one embodiment, this Mongolian-Chinese neural machine translation method uses the BLEU value as an indicator to evaluate the quality of the translation. Its calculation formula is as follows:
[0043]
[0044] p n represents the n-gram exact match rate, w n is the weight of the n-gram, usually taken as N represents the maximum number of n-gram phrases considered when calculating the BLEU value, generally taking N = 4. BP is the length penalty, which is used to prevent the generation of too short translations. Its calculation formula is:
[0045]
[0046] Among them, c is the length of the generated translation, and r is the length of the reference translation.
[0047] The present invention optimizes the supervised fine-tuning by adding two low-rank matrices to capture task-specific knowledge and adding their product to the original checkpoint model weights to adjust the Mongolian-Chinese neural machine translation model, reducing the computational overhead compared to directly adjusting the parameters and avoiding the negative optimization problem. The present invention also constructs prompt engineering, constructs prompts from two dimensions of word meaning and semantics, and utilizes the powerful context understanding ability of the large language model in high-resource languages to assist in improving the effect of low-resource Mongolian translation tasks. The present invention can demonstrate good functions and advantages in the Mongolian-Chinese neural machine translation task, and the optimized model performs excellently in low-resource Mongolian translation, significantly improving the translation quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is the overall framework diagram of the present invention.
[0049] Figure 2 is the low-rank adaptation fine-tuning process diagram.
[0050] Figure 3 is the process of multi-dimensional prompt design. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0051] The following describes the embodiments of the present invention in detail with reference to the drawings and embodiments:
[0052] As Figure 1 shown, the present invention is mainly a Mongolian-Chinese neural machine translation method optimized by multi-dimensional prompts based on a large language model, and the steps are as follows:
[0053] Step 1, supervise and fine-tune a pre-trained large language model M on a large-scale Mongolian-Chinese parallel corpus through the low-rank adaptation method, and specifically optimize the Mongolian-to-Chinese translation task to improve the translation performance. The core idea of this fine-tuning method is to train only a few parameters while maintaining the performance that can be achieved by the traditional full-scale fine-tuning method, thereby reducing the computational cost and avoiding the risk of negative transfer, which is particularly suitable for the optimization of low-resource language pairs.
[0054] Specifically, as Figure 2 shown, the difference between the low-rank adaptation fine-tuning method and the general parameter update fine-tuning is that it does not directly optimize the parameters on the original model parameters. Specifically, for a given pre-trained weight matrix W0 and a weight update matrix ΔW, W0 + ΔW is the fine-tuning parameter update result of LoRA. The partial parameter fine-tuning method of LoRA mainly optimizes two low-rank matrices A and B containing trainable parameters added in certain specific layers during the fine-tuning process. The weight update matrix ΔW is decomposed into the product of these two low-rank matrices A and B, where A and B are matrices of dimension i×j and j×k over the real number field, respectively, and j << min(i, k).
[0055] Step 2, refer to Figure 3 , and construct a semantic chain prompt. Use the TF-IDF method to extract several keywords for each sentence from the Mongolian corpus. The formula is:
[0056]
[0057] where f(t, d) represents the frequency of occurrence of word t in sentence d, N d represents the total number of words in sentence d, N is the total number of sentences in the corpus, and df(t) is the total number of sentences containing word t;
[0058] At the same time, filter through the weight threshold to avoid the interference of the most frequent function words, numbers and other noises, and ensure the semantic relevance and translation value of the extracted keywords.
[0059] Each keyword is translated into corresponding vocabulary in two higher-resource auxiliary languages (such as English and Cyrillic Mongolian) through the BabelNet multilingual knowledge base tool, and merged into a semantic chain prompt. This prompt structure aims to make up for the deficiency of the large language model in the translation ability of low-resource languages through the rich semantic and context information of high-resource languages.
[0060] Step 3, as Figure 3 shown, use the LaBSE semantic embedding model based on the BERT model to embed the sentence through self-supervised learning method, generate language-independent sentence vectors, map x and the entire Chinese target monolingual corpus Dy to the same semantic space. In this space, the model calculates the similarity between the source sentence x and the target sentence y, and selects the K most similar Chinese sentences to the source sentence as translation examples. Use the cosine similarity formula to calculate the similarity score between the Mongolian source sentence vector and all Chinese target sentence vectors. The formula is as follows:
[0061]
[0062] where v is the high-dimensional semantic vector generated by the embedding model; ||v|| is the norm of the vector.
[0063] The selected similar sentences are highly relevant to the Mongolian source sentence to be translated in terms of semantics and syntactic structure, and can help the model better understand the context of the source sentence. To prevent the model from being inclined to retain the source grammar rules under the prompt and thus affecting the translation quality, the target language sentences are selected as examples in the prompt.
[0064] Step 4, analyze the syntactic patterns of the K sentence examples to form a syntactic pattern prompt. Specifically, as Figure 3As shown, by guiding the large language model M to divide different parts of each example in the form of natural language prompts, identify components such as the subject, verb, adverbial phrase, object, etc., and construct a semantic association structure, it is used to guide the model to more accurately capture the complex grammatical structures and semantic alignments in the process of Mongolian to Chinese translation.
[0065] Step 5, design explicit instruction prompts to simply and clearly describe the task goal and directly guide the large language model to perform translation operations.
[0066] Step 6, splice the constructed word meaning chain prompts, syntactic pattern prompts, and explicit instruction prompts, and input them into the large language model M to obtain the final Mongolian-Chinese translation result.
[0067] The Mongolian-Chinese neural machine translation method uses the BLEU value as an indicator to evaluate the quality of the translated text, and its calculation formula is as follows:
[0068]
[0069] p n represents the n-gram exact match rate, w n is the weight of the n-gram, usually taking N represents the maximum number of n-gram phrases considered when calculating the BLEU value, generally taking N = 4, and BP is the length penalty, which is used to prevent the generation of overly short translated texts, and its calculation formula is:
[0070]
[0071] where c is the length of the generated translated text and r is the length of the reference translated text.
[0072] For example, for the Mongolian source sentence (Standard Chinese interpretation: His mother tongue is Spanish), screen several keywords and translate them. Among them, is translated into the corresponding English "language" and Cyrillic Mongolian "Хэл", and thus a word meaning chain prompt is formed. Then find the K Chinese sentences with the highest similarity in the corpus. For example, the sentence "Spanish runs through his daily life" is parsed by the large language model M to obtain that the subject of this sentence is "Spanish", the verb is "runs through", and the object is "daily life", and thus a syntactic pattern prompt is formed. Finally, splice the explicit instruction prompt containing the explicit translation instruction and the current Mongolian source sentence with the previous two prompts and input them into the large language model M for translation, and the high-quality translation result "His mother tongue is Spanish" can be obtained.
Claims
1. A Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of large language models, characterized in that, It includes the following steps: Step 1, supervise and fine-tune a pre-trained large language model M on the Mongolian-Chinese parallel corpus through the low-rank adaptation method; Step 2: Construct prompts from the semantic dimension for each sentence x in the Mongolian corpus i Extract several keywords; translate each keyword into Chinese and corresponding words in two auxiliary languages with higher resources, and combine them into semantic chain prompts; Step 3, for the current Mongolian sentence x to be translated, use the semantic embedding model to map x and the entire Chinese target monolingual corpus Dy into the same semantic space; calculate the similarity scores between x and each sentence in Dy, and obtain the K most similar sentences as examples; Step 4, design instruction prompts to guide the pre-trained large language model M to parse the syntactic patterns of the K examples respectively, and combine them to form syntactic pattern prompts; Step 5, design explicit instruction-type prompts to clearly describe the task objective and directly guide the pre-trained large language model M to perform translation operations; Step 6, for each sentence x i concatenate the corresponding semantic chain prompts, syntactic pattern prompts, and explicit instruction prompts, and input them into the pre-trained large language model M to obtain the Mongolian-Chinese translation result.
2. The Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of large language models according to claim 1, wherein In Step 1, a pre-trained large language model based on the Transformer architecture is adopted, and the self-attention mechanism is used to dynamically capture the semantic associations between each word and other words in the input sequence. The formula is as follows: Among them, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively, which are embedding vectors from the input sequence; d k is the dimension of the key vector, which is used for normalization to stabilize the gradient; In the low-rank adaptation method, two low-rank matrices A and B are added to capture the knowledge of the Mongolian-Chinese translation task. Finally, the product of the two low-rank matrices A and B is added to the original weight matrix of the pre-trained large language model M to complete the fine-tuning of the model.
3. The Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of large language models according to claim 1, characterized in that, In Step 2, the term frequency-inverse document frequency (TF-IDF) method is used for keyword extraction. The formula is as follows: where f(t, d) represents the frequency of occurrence of word t in sentence d, N d represents the total number of words in sentence d, N is the total number of sentences in the corpus, and df(t) is the total number of sentences containing word t; Each keyword is translated into the target Chinese vocabulary and its corresponding two more resource-rich auxiliary language vocabularies through a multilingual knowledge base tool, and merged into a semantic chain prompt.
4. The Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of large language models according to claim 1, characterized in that The format of the semantic chain prompt is as follows: "{The following gives the keyword and its translations in different languages:} ‘<Mongolian word X source >’ corresponds to the Chinese interpretation of ‘<Chinese corresponding word X target >’ Equivalent paraphrase as '<corresponding word X1 in auxiliary language 1>', Equivalent paraphrase as '<corresponding word X2 in auxiliary language 2>'".
5. The Mongolian-Chinese neural machine translation method optimized by multi-dimensional prompts based on the large language model according to claim 1, wherein In Step 3, the cosine similarity formula is used to calculate the similarity scores between the Mongolian source sentence vector and all Chinese target sentence vectors. The formula is as follows: where v mo and v zh are the high-dimensional semantic vectors of Mongolian and Chinese generated by the embedding model respectively; |||| represents the norm of the vector.
6. The Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of large language models according to claim 1, wherein In Step 4, the instruction prompt content is a natural language instruction. The pre-trained large language model M predicts the part-of-speech of each word in the sentence according to the instruction, combined with the context information and the language knowledge it has learned. The instruction content is as follows: "For the given set of Mongolian-Chinese parallel examples, parse the K Chinese examples into syntactic patterns" The results are output in the form of standard part-of-speech tags. The syntactic analysis format of each Chinese sentence is as follows: "{Syntactic pattern:} Subject: <subject>, Verb: <verb>, Object: <object>, Time / adverbial phrase: <time / adverbial phrase>, Subordinate clause subject: <subordinate clause subject>, Subordinate clause verb: <subordinate clause verb>, Subordinate clause object: <subordinate clause object>" Among them, <...> represents the corresponding vocabulary in the sentence. After further sorting, the formed syntactic pattern prompt format is as follows: "{Given K example sentences and their Chinese syntactic patterns:} {Example 1:} <Chinese sentence 1> {Syntactic pattern 1:} <syntactic analysis 1> ... {Example K:} <Chinese sentence K> {Syntactic pattern K:} <syntactic analysis K>".
7. The Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of large language models according to claim 1, characterized in that In Step 5, the explicit instruction-type prompt clearly guides the pre-trained large language model M to perform the correct task. The prompt content is as follows: "Please translate the following Mongolian sentence into Chinese: Mongolian: <Mongolian sentence> Chinese: " Among them, {Chinese:} is the flag for the pre-trained large language model M to start performing translation.
8. The Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of large language models according to claim 1, characterized in that In step 6, for each sentence x to be translated i The text content of the corresponding semantic chain prompt, syntactic pattern prompt, and explicit instruction prompt is concatenated and combined in sequence through Python code to form the overall input content of the sentence; Subsequently, the integrated prompt is input into the pre-trained large language model M to guide the model to generate a more accurate translation output.
9. The Mongolian-Chinese neural machine translation method based on multi-dimensional prompt optimization of large language models according to claim 1, characterized in that, Use the BLEU value as an indicator to evaluate the quality of the translation, and its calculation formula is as follows: p n represents the n-gram exact match rate, w n is the weight of the n-gram, taking N represents the maximum number of n-gram phrases considered when calculating the BLEU value, taking N = 4, and BP is the length penalty used to prevent the generation of overly short translations. Its calculation formula is: Among them, c is the length of the generated translation, and r is the length of the reference translation.