MoE and representation decoupled based large model multilingual translation method and device

Abstract

This invention relates to a large-scale multilingual translation method and apparatus based on MoE and representation decoupling, belonging to the field of natural language processing technology. The Chinese-English-Lao-Thai multilingual translation framework can effectively share similar language knowledge and improve translation performance in different translation directions through model parameter sharing. However, introducing parameter sharing can lead to parameter interference between different languages. To address the parameter interference problem between different languages ​​in multilingual translation, this invention is proposed. The invention comprises three parts: data preprocessing, training a large-scale multilingual translation model based on MoE and representation decoupling, performing translation using the trained large-scale multilingual translation model based on MoE and representation decoupling, and an apparatus composed of modules integrated from these three parts. This invention combines a sparse MoE framework and representation decoupling to alleviate the multilingual parameter interference problem, effectively improving the quality of Chinese-English-Lao-Thai multilingual machine translation.

Description

Technical Field

[0001] This invention relates to a method and apparatus for multilingual translation of large models based on MoE and representation decoupling, belonging to the field of natural language processing technology. Background Technology

[0002] Machine translation technology can effectively reduce the cost of language communication, bridging speakers of different languages ​​and facilitating cross-cultural communication and cooperation. Compared with traditional human translation, machine translation has advantages in efficiency, low cost, and scalability, making it particularly significant in multilingual communication scenarios. Current neural machine translation performance relies on the scale and quality of bilingual parallel corpora, achieving good results in translating large-scale parallel corpora, but still falls short in low-resource translation tasks such as Lao / Thai-Chinese. Multilingual machine translation methods: Thai and Lao are typical low-resource languages, and current research on neural machine translation methods for low-resource languages ​​has been extensive. Among these, multilingual neural machine translation is an effective means to address the low-resource problem. This method achieves the translation of multiple language pairs within a single model, leveraging translation knowledge from high-resource language pairs to improve the translation performance of low-resource language pairs, while also reducing the computational resource consumption for model training and deployment. Knowledge transfer in multilingual machine translation comes from shared vocabularies and shared model parameters; word embedding knowledge can be transferred through cognates in the shared vocabulary.

[0003] However, although Thai and Lao belong to the same Tai-Kadai language family and share a high degree of linguistic affinity, their character writing systems and Romanization rules differ significantly. As shown in Table 1, Thai and Lao characters with the same meaning exhibit clear differences in morphology and transliteration. This superficial similarity can easily lead to cross-linguistic interference within a shared representation space. Hybrid expert systems (MoEs), through expert network parameter isolation and dynamic routing mechanisms, naturally possess the characteristics to mitigate interference between multiple tasks.

[0004] Table 1 shows the differences in Romanization between Thai and Lao.

[0005]

[0006] To address this, the present invention proposes a large-scale multilingual translation method based on MoE and representation decoupling. Summary of the Invention

[0007] The technical problem to be solved by this invention is that it provides a large-scale multilingual translation method and apparatus based on MoE and representation decoupling to solve the parameter interference problem between different languages ​​in multilingual translation. This invention combines a sparse MoE framework and representation decoupling to alleviate the multilingual parameter interference problem and effectively improve the quality of Chinese-English-Lao-Thai multilingual machine translation.

[0008] The technical solution of this invention is: a large-scale multilingual translation method based on MoE and representation decoupling, the method comprising:

[0009] Step 1, Data Preprocessing: First, construct the instruction dataset required for model training;

[0010] Step 2: Construct a large-scale multilingual translation model based on MoE and representation decoupling, and then use the instruction dataset to train the large-scale multilingual translation model based on MoE and representation decoupling:

[0011] First, a sparse MoE architecture is used to build different decoding modules for English-Chinese, Lao-Chinese, and Thai-Chinese translation directions;

[0012] At the same time, character-level contrastive learning constraints are introduced to separate the hidden layer representations of different translation directions in the model, thereby achieving the effect of parameter decoupling;

[0013] Then, based on the instruction dataset constructed in Step 1, a large-scale multilingual translation model based on MoE and decoupling of representation is obtained;

[0014] Step 3: Use the trained large-scale multilingual translation model based on MoE and representation decoupling to perform translation.

[0015] Furthermore, Step 1 includes the following steps:

[0016] Step 1.1 Data Construction: The original English-Chinese, Lao-Chinese, and Thai-Chinese parallel translation corpora were initially cleaned using the DeepSeek large language model;

[0017] Specifically, the prompt tells the DeepSeek large language model to filter out raw translation corpora with fewer than 20 characters and that are not parallel, in order to extract high-quality parallel translation corpora from the messy data and initially build high-quality parallel translation data.

[0018] Step 1.2 Data Processing: The high-quality parallel translation corpus obtained in Step 1.1 is segmented and regularized to remove English entities and translation bias phrases from the corpus, and the corpus is further filtered.

[0019] Step 1.3 Instruction Construction: Use script tools to convert the high-quality translation corpus obtained in Step 1.2 into a format, construct the translation instruction set required for subsequent large-scale model training, and divide the translation instruction set into training instruction set, verification instruction set and test instruction set.

[0020] Furthermore, Step 2 includes the following steps:

[0021] Step 2.1: Select Llama3-8b-it as the base model, and add a hybrid expert layer to the feedforward neural network layer of the base model;

[0022] The HydraLoRA low-rank adapter with an asymmetric structure replaces the linear layers in the feedforward neural network layer of the base model; decoder modules are designed for English-Chinese, Lao-Chinese, and Thai-Chinese translation to collaboratively solve translation tasks in different directions.

[0023] The forward propagation process of the feedforward neural network layer in the Transformer architecture can be simplified as follows:

[0024]

[0025] in, Sentence vector representation in the model. This represents a feedforward neural network layer. Represent a transformation function;

[0026] The matrix operations of the linear layer in the forward propagation are represented as follows:

[0027]

[0028] in, This represents the parameter matrix in the basic model. This indicates parameter updates during the training phase. N Indicates the number of experts. This represents the importance coefficient of the k-th expert; A represents the parameter matrix associated with the k-th expert; A represents the transformation matrix.

[0029] Step 2.2: Design a character-level contrastive learning mechanism. By introducing character-level contrastive learning constraints into a hybrid expert layer, different experts are forced to learn a more discriminative language representation space; including the following:

[0030] In the hybrid expert layer of the feedforward neural network, a character-level contrastive learning loss function is defined. as follows:

[0031]

[0032] in, Indicates the first k An expert on the i-th token Hidden layer representation. Indicates the first An expert on the i-th token Hidden layer representation. This indicates that the m-th expert has a view on the j-th token. The hidden layer representation, s(·,·) is the cosine similarity function: , r ∈[﹣1,1] is the boundary threshold, and token represents a word element after word segmentation;

[0033] Step 2.3: In the basic model training stage, soft constraints are imposed by designing task types to ensure that experts focus on their respective translation tasks and maintain balance among experts on similar tasks.

[0034] Define the importance matrix of the hybrid expert layer Q ∈R N×M ,in Q n,m Indicates the first in the batch m The training sample pair for the th training sample n The sum of router weights for each expert:

[0035]

[0036] in, N and T m Representing the number of experts and the number of... m The number of tokens in each sample For the j-th token to the th n Routing scores of experts, For the j-th token to the th k Routing scores of experts, This is a temperature parameter used to control the "sharpness" or "smoothness" of the softmax function;

[0037] Define the coefficient matrix I ∈R N×M :

[0038]

[0039] in, d ∈[0,1] controls the degree of imbalance between expert types. and They represent experts respectively. n Target type and sample m Task type; experts satisfying similar tasks I a,k1 = I b,k1 Different types of experts meet the requirements. I a,k1 ≠ I b,k1 Here, a and b represent different experts, while k1 represents the task type. Ia,k1 Represents the coefficient matrix I The element in the a-th row and k1-th column, I b,k1 Represents the coefficient matrix I The element in the b-th row and k1-th column;

[0040] Load balancing losses Defined as:

[0041]

[0042] in, Represents the Hadamard product. s 2 (·)and m (·) Calculate the variance and mean respectively;

[0043] Step 2.4: Combine cross-entropy loss Combined with load balancing loss, design the total loss function of the model. Defined as:

[0044]

[0045] Among them, hyperparameters c Controlling the intensity of contrastive learning, It is a hyperparameter used to adjust the weight of load balancing loss in the total loss;

[0046] Synchronous optimization via gradient backpropagation:

[0047]

[0048] in, This represents all the parameters in the model that need to be learned and optimized;

[0049] Step 2.5: Freeze the parameters of the base model, and use the instruction dataset constructed in Step 1 to train the constructed large-scale multilingual translation model based on MoE and representation decoupling, and make the model reach convergence again; finally, the trained large-scale multilingual translation model based on MoE and representation decoupling is obtained.

[0050] Furthermore, in Step 3: First, the large-scale multilingual translation model based on MoE and representation decoupling obtained in Step 2 is encapsulated into an API interface that can be called on the server;

[0051] Then, a multi-terminal visualization platform for Chinese-English-Lao-Thai multilingual translation models is built.

[0052] Finally, the Chinese-English-Lao-Thai multilingual translation visualization platform obtains the corresponding translation results by calling the API interface deployed on the server.

[0053] Furthermore, Step 3 includes the following steps:

[0054] Step 3.1: Merge the weights of the trained large model and use the LMDepoy large model quantization deployment tool to deploy the large multilingual translation model based on MoE and decoupling of representation to the server. By calling the model, the function of multi-user concurrent requests can be realized.

[0055] Step 3.2: The user specifies the text to be translated and the target language through the front-end graphical user interface (GUI) and clicks send. The input text information is sent to the server through the API interface. Finally, the translation result is obtained by reasoning based on the large-scale multilingual translation model that decouples MoE and representation and is returned to the front end.

[0056] User input is English, Lao, or Thai text. After preprocessing, the text is converted into a format that the model can understand. The trained large-scale multilingual translation model based on MoE and decoupling of representation generates high-quality translation results according to the input text language and the target language.

[0057] The present invention also provides a large-scale multilingual translation apparatus based on MoE and representation decoupling, the apparatus comprising: for performing the large-scale multilingual translation method based on MoE and representation decoupling.

[0058] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large model multilingual translation method based on MoE and representation decoupling.

[0059] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the large-model multilingual translation method based on MoE and representation decoupling.

[0060] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the large-model multilingual translation method based on MoE and representation decoupling.

[0061] The beneficial effects of this invention are:

[0062] 1. This invention is based on the sparse MoE framework and uses a language-specific lightweight LoRA module to achieve domain-specific modeling of English / Lao / Thai-Chinese multilingual pairs. Each LoRA expert only fine-tunes the parameters for a specific language direction. At the same time, a task-aware dynamic routing mechanism is designed to adaptively allocate expert weights according to the source → target language combination during training, thereby suppressing parameter coupling between different language directions.

[0063] 2. This invention introduces character-level contrastive learning constraints. For bilingual character pairs with the same semantics, negative sample contrastive learning constraints are used to maximize the hidden distance of negative sample pairs and enhance the representation consistency of character variants within the same language through positive sample alignment. This decouples shallow similarity interference between languages, allowing the model to focus on deep semantic associations and achieve more accurate mapping on low-resource language pairs, thereby enhancing the performance of different translation directions.

[0064] 3. This invention combines a sparse MoE framework and representation decoupling to alleviate the problem of multilingual parameter interference, thereby improving the quality of multilingual machine translation between Chinese, English, Lao, and Thai. Attached Figure Description

[0065] Figure 1 This is an architecture diagram of a large-scale multilingual translation method based on MoE and representation decoupling;

[0066] Figure 2 This is a flowchart of the method in this invention. Detailed Implementation

[0067] Example 1: A method and apparatus for large-scale multilingual translation based on MoE and representation decoupling, the method comprising:

[0068] Step 1, Data Preprocessing: First, construct the instruction dataset required for model training;

[0069] Step 2: Construct a large-scale multilingual translation model based on MoE and representation decoupling, and then use the instruction dataset to train the large-scale multilingual translation model based on MoE and representation decoupling:

[0070] First, a sparse MoE (Mixture of Experts) architecture is used to build different decoding modules for the three translation directions: English-Chinese, Lao-Chinese, and Thai-Chinese.

[0071] At the same time, character-level contrastive learning constraints are introduced to separate the hidden layer representations of different translation directions in the model, thereby achieving the effect of parameter decoupling;

[0072] Then, based on the instruction dataset constructed in Step 1, a large-scale multilingual translation model based on MoE and decoupling of representation is obtained;

[0073] Step 3: Use the trained large-scale multilingual translation model based on MoE and representation decoupling to perform translation.

[0074] Furthermore, Step 1 includes the following steps:

[0075] Step 1.1 Data Construction: The original English-Chinese, Lao-Chinese, and Thai-Chinese parallel translation corpora were initially cleaned using the DeepSeek large language model;

[0076] Specifically, the prompt tells the DeepSeek large language model to filter out raw translation corpora with fewer than 20 characters and that are not parallel, in order to extract high-quality parallel translation corpora from the messy data and initially build high-quality parallel translation data.

[0077] Step 1.2 Data Processing: The high-quality parallel translation corpus obtained in Step 1.1 is segmented and regularized to remove English entities and translation bias phrases from the corpus, further filtering the corpus and laying a high-quality data foundation for subsequent method proofs;

[0078] Step 1.3 Instruction Construction: Use script tools to convert the high-quality translation corpus obtained in Step 1.2 into a format, construct the translation instruction set required for subsequent large-scale model training, and divide the translation instruction set into training instruction set, verification instruction set and test instruction set.

[0079] Furthermore, Step 2 includes the following steps:

[0080] Step 2.1: Select Llama3-8b-it as the base model, and add a hybrid expert layer to the feedforward neural network layer of the base model;

[0081] The HydraLoRA low-rank adapter with an asymmetric structure replaces the linear layers in the feedforward neural network layer of the base model; decoder modules are designed for English-Chinese, Lao-Chinese, and Thai-Chinese translation to collaboratively solve translation tasks in different directions.

[0082] The forward propagation process of the feedforward neural network layer in the Transformer architecture can be simplified as follows:

[0083]

[0084] in, Sentence vector representation in the model. This represents a feedforward neural network layer. Represent a transformation function;

[0085] The matrix operations of the linear layer in the forward propagation are represented as follows:

[0086]

[0087] in, This represents the parameter matrix in the basic model. This indicates parameter updates during the training phase. N Indicates the number of experts. This represents the importance coefficient of the k-th expert; A represents the parameter matrix associated with the k-th expert; A represents the transformation matrix.

[0088] Step 2.2 addresses the problem in multilingual translation tasks where competitive parameter sharing leads to the suppression or even overwriting of representations for some language pairs. This invention designs a character-level contrastive learning mechanism. By introducing character-level contrastive learning constraints into a hybrid expert layer, it forces different experts to learn a more discriminative language representation space. The core of this method lies in decoupling feature entanglement among experts and alleviating representation conflicts caused by parameter sharing; it includes the following:

[0089] In the hybrid expert layer of the feedforward neural network, a character-level contrastive learning loss function is defined. as follows:

[0090]

[0091] in, Indicates the first k An expert on the i-th token Hidden layer representation. Indicates the first An expert on the i-th token Hidden layer representation. This indicates that the m-th expert has a view on the j-th token. The hidden layer representation, s(·,·) is the cosine similarity function: , r ∈[﹣1,1] is the boundary threshold, and token represents a word element after word segmentation;

[0092] Step 2.3: In the basic model training stage, soft constraints are imposed by designing task types to ensure that experts focus on their respective translation tasks and maintain balance among experts on similar tasks.

[0093] Define the importance matrix of the hybrid expert layer Q ∈R N×M ,in Q n,m Indicates the first in the batch m The training sample pair for the th training sample n The sum of router weights for each expert:

[0094]

[0095] in, N and T m Representing the number of experts and the number of... m The number of tokens in each sample For the j-th token to the th n Routing scores of experts, Let j be the routing score between the j-th token and the k-th expert. This is a temperature parameter used to control the "sharpness" or "smoothness" of the softmax function;

[0096] Define the coefficient matrix I ∈R N×M :

[0097]

[0098] in, d ∈[0,1] controls the degree of imbalance between expert types. and They represent experts respectively. n Target type and sample m Task type; experts satisfying similar tasks I a,k1 = I b,k1 Different types of experts meet the requirements. I a,k1 ≠ I b,k1 Here, a and b represent different experts, while k1 represents the task type. I a,k1 Represents the coefficient matrix I The element in the a-th row and k1-th column, I b,k1 Represents the coefficient matrix I The element in the b-th row and k1-th column;

[0099] Load balancing losses Defined as:

[0100]

[0101] in, Represents the Hadamard product. s 2 (·)and m (·) Calculate the variance and mean respectively;

[0102] Step 2.4: Combine cross-entropy loss Combined with load balancing loss, design the total loss function of the model. Defined as:

[0103]

[0104] Among them, hyperparameters c Controlling the intensity of contrastive learning, It is a hyperparameter used to adjust the weight of load balancing loss in the total loss;

[0105] Synchronous optimization via gradient backpropagation:

[0106]

[0107] in, This represents all the parameters in the model that need to be learned and optimized;

[0108] Step 2.5: Freeze the parameters of the base model, and use the instruction dataset constructed in Step 1 to train the constructed large-scale multilingual translation model based on MoE and representation decoupling, and make the model reach convergence again; finally, the trained large-scale multilingual translation model based on MoE and representation decoupling is obtained.

[0109] Furthermore, in Step 3: First, the large-scale multilingual translation model based on MoE and representation decoupling obtained in Step 2 is encapsulated into an API interface that can be called on the server;

[0110] Then, a multi-terminal visualization platform for Chinese-English-Lao-Thai multilingual translation models is built.

[0111] Finally, the Chinese-English-Lao-Thai multilingual translation visualization platform obtains the corresponding translation results by calling the API interface deployed on the server.

[0112] Furthermore, Step 3 includes the following steps:

[0113] Step 3.1: Merge the weights of the trained large model and use the LMDepoy large model quantization deployment tool to deploy the large multilingual translation model based on MoE and decoupling of representation to the server. By calling the model, the function of multi-user concurrent requests can be realized.

[0114] Step 3.2: The user specifies the text to be translated and the target language through the front-end graphical user interface (GUI) and clicks send. The input text information is sent to the server through the API interface. Finally, the translation result is obtained by reasoning based on the large-scale multilingual translation model that decouples MoE and representation and is returned to the front end.

[0115] User input is English, Lao, or Thai text. After preprocessing, the text is converted into a format that the model can understand. The trained large-scale multilingual translation model based on MoE and decoupling of representation generates high-quality translation results according to the input text language and the target language.

[0116] The present invention also provides a large-scale multilingual translation apparatus based on MoE and representation decoupling, the apparatus comprising: for performing the large-scale multilingual translation method based on MoE and representation decoupling.

[0117] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large model multilingual translation method based on MoE and representation decoupling.

[0118] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the large-model multilingual translation method based on MoE and representation decoupling.

[0119] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the large-model multilingual translation method based on MoE and representation decoupling.

[0120] To verify the effectiveness of the proposed method, experiments were conducted on a Linux system. Supervised fine-tuning of the pre-trained Llama3 series models was performed using the improved HydraLoRA framework, employing a lightweight LoRA adaptation method. Specific parameter configurations included: distributed training using four GPUs, a single-GPU training batch size of 1, gradient accumulation steps of 4 steps, an equivalent total batch size of 20, a maximum sequence length limit of 512, LoRA module parameters set to rank (rank=4), alpha value of 32, random inactivation rate of 0.05, and target modules limited to gate_proj, down_proj, and up_proj, with three independent LoRA branches enabled. The optimizer used was AdamW, with an initial learning rate of 2×10⁻⁶. -4The learning rate was adjusted using a cosine annealing scheduler, with a warm-up ratio of 3%, and weight decay and gradient clipping were disabled. FP16 mixed precision and Flash Attention were enabled for acceleration during training, and model parameters were quantized to 16 bits. DeepSpeed ​​Zero stage 3 was used to optimize GPU memory, and `max_split_size_mb=128` was set to reduce memory fragmentation. The total number of training epochs was 5, with a logging interval of 10 steps and a model save / validation interval of 5000 steps, and early stopping was enabled. Distributed training timeout was set to 30000 seconds, and CUDA synchronization was forced to aid in debugging. The BLEU metric was used to evaluate model performance, effectively determining the text similarity between the model-generated translation and the reference translation.

[0121] Experiment 1: Comparison of Experimental Results of Different Models in One-to-One and Many-to-One Translation Scenarios

[0122] In the experiments, the model was compared with other models based on the Fairseq framework, and the best results were analyzed. The comparative experiments were set up as follows:

[0123] (1) Transformer: Bilingual neural machine translation between Chinese and Thai in one-to-one scenarios, using the standard Transformer structure.

[0124] (2) Multilingual: The standard Transformer multilingual neural machine translation model method from Thai and Lao to Chinese.

[0125] (3) CIAT Adapter: Based on Multilingual, it uses the adapter module proposed by ByteDance to alleviate multilingual interference for fine-tuning.

[0126] (4) Llama 3-it-zero-shot: Use the Llama3 instruction version model to perform zero-shot to realize multilingual translation tasks.

[0127] (5) Llama 3-it-LoRA: Use the Llama3 instruction version model to perform joint training of a single LoRA to achieve multilingual translation tasks.

[0128] Table 2 shows the BLEU experimental results in one-to-one and many-to-one translation scenarios.

[0129]

[0130] As shown in Table 2, the proposed method achieves good performance on both low-resource language pairs (Lao→Chinese, Thai→Chinese) and high-resource language pairs (English→Chinese), validating its effectiveness. For the Lao→Chinese task, the proposed method improves BLEU by 0.88 compared to the traditional multilingual training method and significantly outperforms the CIATAdapter method, indicating that the task isolation mechanism of the MoE architecture can more effectively alleviate the representation confusion between Lao and Thai. In the Thai→Chinese task, although Llama 3-it-LoRA has already achieved a high baseline of 20.30 BLEU through parameter fine-tuning, the proposed method further optimizes it through comparative decoupling, demonstrating that subtle differences between similar language pairs require explicit constraints rather than simple parameter isolation. Notably, Llama 3-it-zero_shot performs poorly on the Lao→Chinese task (0.62 BLEU), but significantly outperforms traditional models on the English→Chinese task, revealing that the zero-shot generalization ability of large models is severely limited on low-resource languages, requiring specific mechanisms to guide knowledge transfer.

[0131] Compared to other methods, the traditional multilingual joint training method outperforms the monolingual Transformer on low-resource language pairs, but the performance gap with the method of this invention (Lao→Chinese: 0.88 BLEU, Thai→Chinese: 1.99 BLEU) reflects the limitations of implicit parameter sharing in multi-task joint training. While the CIAT Adapter mitigates interference through adversarial training, its effectiveness is still limited by the fixed structure of the adapter module, making it difficult to dynamically adapt to the conflict levels of different language pairs. The slight advantage (+0.17 BLEU) of the method of this invention over Llama 3-it-LoRA on the English→Chinese task indicates that even in high-resource scenarios, the dynamic routing mechanism can still improve model focus by suppressing irrelevant expert interference. In summary, the experimental results confirm the synergistic effect of representation decoupling and task-aware routing: the former eliminates semantic ambiguity of similar-looking characters in the representation space, while the latter enables targeted activation of language pair-specific knowledge in the parameter space. Together, they enable the model to maintain better generalization performance in complex multilingual scenarios.

[0132] Experiment 2: Ablation Experiment

[0133] Table 3 shows the ablation experiments.

[0134]

[0135] As shown in Table 3, removing either the Hybrid Expert (MoE) or contrastive learning leads to a decrease in model performance, with the most significant degradation occurring when both are removed simultaneously, validating their synergistic enhancement effect in low-resource language pairs. For the Lao-Chinese task, removing MoE resulted in a BLEU reduction of 0.83, exceeding the 0.68 reduction from contrastive learning removal, indicating that the model relies more heavily on expert isolation to avoid interference from Thai knowledge. In the Thai-Chinese task, contrastive learning removal only reduced BLEU by 0.16, suggesting that the semantic mapping between Thai and Chinese is relatively independent, and character-level confusion is weaker than in Lao. Notably, when both mechanisms are removed simultaneously, the model still maintains 37.14 BLEU in the English-Chinese task, matching the performance of the base Llama 3-it-LoRA, ensuring that representation learning for high-resource language pairs relies more on the capabilities of the base model itself and is less sensitive to decoupling mechanisms. Furthermore, ablation experiments reveal that MoE and contrastive learning are complementary: the former primarily suppresses task conflicts in the parameter space, while the latter focuses on resolving semantic confusion in the representation space.

[0136] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A large-scale multilingual translation method based on MoE and representation decoupling, characterized by: The method includes: Step 1, Data Preprocessing: First, construct the instruction dataset required for model training; Step 2: Construct a large-scale multilingual translation model based on MoE and representation decoupling, and then use the instruction dataset to train the large-scale multilingual translation model based on MoE and representation decoupling: First, a sparse MoE architecture is used to build different decoding modules for English-Chinese, Lao-Chinese, and Thai-Chinese translation directions; At the same time, character-level contrastive learning constraints are introduced to separate the hidden layer representations of different translation directions in the model, thereby achieving the effect of parameter decoupling; Then, based on the instruction dataset constructed in Step 1, a large-scale multilingual translation model based on MoE and decoupling of representation is obtained; Step 3: Use the trained large-scale multilingual translation model based on MoE and representation decoupling to perform translation; The steps in Step 2 include: Step 2.1: Select Llama3-8b-it as the base model, and add a hybrid expert layer to the feedforward neural network layer of the base model; The HydraLoRA low-rank adapter with an asymmetric structure replaces the linear layers in the feedforward neural network layer of the base model; decoder modules are designed for English-Chinese, Lao-Chinese, and Thai-Chinese translation to collaboratively solve translation tasks in different directions. The forward propagation process of the feedforward neural network layer in the Transformer architecture can be simplified as follows: ； in, Sentence vector representation in the model This represents a feedforward neural network layer. Represent a transformation function; The matrix operations of the linear layer in the forward propagation are represented as follows: ； in, This represents the parameter matrix in the basic model. This indicates parameter updates during the training phase. N Indicates the number of experts. This represents the importance coefficient of the k-th expert; A represents the parameter matrix associated with the k-th expert; A represents the transformation matrix. Step 2.2: Design a character-level contrastive learning mechanism. By introducing character-level contrastive learning constraints into a hybrid expert layer, different experts are forced to learn a more discriminative language representation space; including the following: In the hybrid expert layer of the feedforward neural network, a character-level contrastive learning loss function is defined. as follows: ； in, Indicates the first k An expert on the i-th token Hidden layer representation. Indicates the first An expert on the i-th token Hidden layer representation. This indicates that the m-th expert has a view on the j-th token. The hidden layer representation, s(·,·) is the cosine similarity function: , ρ ∈[﹣1,1] is the boundary threshold, and token represents a word element after word segmentation.

2. The large-scale multilingual translation method based on MoE and representation decoupling as described in claim 1, characterized in that: The steps in Step 1 include: Step 1.1 Data Construction: The original English-Chinese, Lao-Chinese, and Thai-Chinese parallel translation corpora were initially cleaned using the DeepSeek large language model; Specifically, the prompt tells the DeepSeek large language model to filter out raw translation corpora with fewer than 20 characters and that are not parallel, in order to extract parallel translation corpora from the messy data and initially construct parallel translation data; Step 1.2 Data Processing: The parallel translation corpus obtained in Step 1.1 is segmented and regularized to remove English entities and translation deviation phrases from the corpus and filter the corpus; Step 1.3 Instruction Construction: Use script tools to convert the format of the translation corpus obtained in Step 1.2, construct the translation instruction set required for subsequent large model training, and divide the translation instruction set into training instruction set, verification instruction set and test instruction set.

3. The large-scale multilingual translation method based on MoE and representation decoupling as described in claim 1, characterized in that: Step 2 also includes: Step 2.3: In the basic model training stage, soft constraints are imposed by designing task types to ensure that experts focus on their respective translation tasks and maintain balance among experts on similar tasks. Define the importance matrix of the hybrid expert layer Q ∈R N×M ,in Q n,m Indicates the first in the batch m The training sample pair for the th training sample n The sum of router weights for each expert: ； in, N and T m Representing the number of experts and the number of... m The number of tokens in each sample For the j-th token to the th n Routing scores of experts Let j be the routing score between the j-th token and the k-th expert. This is a temperature parameter used to control the "sharpness" or "smoothness" of the softmax function; Define the coefficient matrix I ∈R N×M : ； in, δ ∈[0,1] controls the degree of imbalance between expert types. and They represent experts respectively. n Target type and sample m Task type; experts satisfying similar tasks I a,k1 = I b,k1 Different types of experts meet the requirements. I a,k1 ≠ I b,k1 Here, a and b represent different experts, while k1 represents the task type. I a,k1 Represents the coefficient matrix I The element in the a-th row and k1-th column, I b,k1 Represents the coefficient matrix I The element in the b-th row and k1-th column; Load balancing losses Defined as: ； in, Represents the Hadamard product. σ 2 (·)and μ (·) Calculate the variance and mean respectively; Step 2.4: Combine cross-entropy loss Combined with load balancing loss, design the total loss function of the model. Defined as: ； Among them, hyperparameters γ Controlling the intensity of contrastive learning, It is a hyperparameter used to adjust the weight of load balancing loss in the total loss; Synchronous optimization via gradient backpropagation: ； in, This represents all the parameters in the model that need to be learned and optimized; Step 2.5: Freeze the parameters of the base model, and use the instruction dataset constructed in Step 1 to train the constructed large-scale multilingual translation model based on MoE and representation decoupling, and make the model reach convergence again; finally, the trained large-scale multilingual translation model based on MoE and representation decoupling is obtained.

4. The large-scale multilingual translation method based on MoE and representation decoupling according to claim 1, characterized in that: In Step 3: First, the large-scale multilingual translation model based on MoE and representation decoupling obtained in Step 2 is encapsulated into an API interface that can be called on the server. Then, a multi-terminal visualization platform for Chinese-English-Lao-Thai multilingual translation models is built. Finally, the Chinese-English-Lao-Thai multilingual translation visualization platform obtains the corresponding translation results by calling the API interface deployed on the server.

5. The large-scale multilingual translation method based on MoE and representation decoupling according to claim 1, characterized in that: The steps in Step 3 include: Step 3.1: Merge the weights of the trained large model and use the LMDepoy large model quantization deployment tool to deploy the large multilingual translation model based on MoE and decoupling of representation to the server. By calling the model, the function of multi-user concurrent requests can be realized. Step 3.2: After specifying the text to be translated and the target language through the front-end graphical user interface (GUI), the user clicks "send" and sends the input text information to the server through the API interface. Finally, the translation result is obtained by reasoning based on the large-scale multilingual translation model that decouples MoE and representation and is returned to the front end. User input is English, Lao, or Thai text. After preprocessing, the text is converted into a format that the model can understand. The trained large-scale multilingual translation model based on MoE and decoupling of representation generates translation results according to the input text language and the target language.

6. A large-scale multilingual translation device based on MoE and representation decoupling, characterized in that, The apparatus includes: a method for performing large-model multilingual translation based on MoE and representation decoupling as described in any one of claims 1 to 5.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the large-model multilingual translation method based on MoE and representation decoupling as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the large-model multilingual translation method based on MoE and representation decoupling as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the large-model multilingual translation method based on MoE and representation decoupling as described in any one of claims 1 to 5.

Images