Large-model multi-language enhancement training method capable of continuously expanding experts

Abstract

The invention provides a large-model multi-language enhancement training method capable of continuously expanding experts, which comprises the following steps: step 1, extracting an original FFN module in a large-scale language model LLM to be trained and segmenting the original FFN module into a plurality of new fine-grained FFN modules as fine-grained experts; 2, setting a fine-grained router, and selecting a preset number of newly-added fine-grained experts according to a preset rule to obtain the output of the fine-grained router; 3, setting a parent router, controlling the weight between the output of the original FFN module and the output of the fine-grained router, and calculating the final output; 4, freezing an original FFN module, and independently training the fine-grained expert; step 5, freezing an original FFN module, adding the newly added fine-grained expert, and training the newly added fine-grained expert; step 6, freezing all other parameters, and only training a parent router; and 7, continuously repeating the step 4 to the step 6 to realize large-model multi-language enhancement training of sustainable extension experts.

Description

Technical Field

[0001] The invention relates to a large-model multi-language enhanced training method, in particular to a large-model multi-language enhanced training method that can sustainably expand experts. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] With the advent of the deep learning era, large language models (LLMs), such as GPT-4 (reference: Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report [J]. arXiv preprint arXiv: 2303.08774, 2023.) and Qwen (reference: Bai J, Bai S, Chu Y, et al. Qwen technical report [J]. arXiv preprint arXiv: 2309.16609, 2023.), rely on large-scale multilingual datasets for pre-training to master the ability to understand, generate, and translate various human languages. Although these models perform well on multiple languages, they still have difficulties in processing some low-resource languages. Traditional model training methods mainly use supervised learning, which requires a large amount of labeled data, which is difficult to achieve in the case of low-resource languages. Therefore, academia and industry have explored various methods to enhance the performance of LLMs on low-resource languages.

[0004] Mixture of Experts (MoE) is a special ensemble learning strategy, whose model architecture usually consists of an overall skeleton, several "expert" models, and a "router". The advantage of MoE is that the expert models can be trained on different data separately to cover more knowledge areas. At the same time, the calculation process of the experts does not affect each other, and will not increase too much reasoning calculation overhead. The core idea of ​​MoE is that when given input, only some experts are activated and trained instead of all experts, so as to reduce the calculation overhead while ensuring a large model scale. At present, MoE technology is usually used in the Transformer model, taking the forward feedback module of each layer as its expert, combined with its unique routing mechanism to form a MoE module.

[0005] In the prior art, MoE-LPR (reference: Zhou H, Wang Z, Huang S, et al. MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Expertswith Language Priors Routing[J].arXiv preprint arXiv:2408.11396,2024.) adopts a two-stage training method to enhance multilingual capabilities. Figure 1 As shown in the figure, the scheme first pre-trains the model by copying the FFN network into a mixture of experts (MoE) architecture, at which time all the original parameters are frozen and new experts are added. At this stage, MoE-LPR focuses on improving the ability of the extended language without using any original language data. Subsequently, the model uses a small amount of (150,000 data) of playback data to help the model review the original language knowledge. Incorporating language prior knowledge at this stage helps the router have a clearer understanding of the original language so as to better restore the ability of the original language.

[0006] However, the implementation of MoE-LPR technology faces several problems:

[0007] High computing resource requirements: Post-pre-training technology requires replicating the FFN network, resulting in too many training parameters, high expansion costs, and very high computational overhead for training the model.

[0008] Risk of catastrophic forgetting: Language expansion will bring about serious distribution shifts. When MoE-LPR focuses on new languages, the forgetting of high-resource language knowledge still exists, affecting the overall performance of the model.

[0009] Insufficient differentiation of experts: When MoE-LPR is expanded, the expert initialization parameters are consistent with the FFN (feedforward neural network) module of the original model, which makes it difficult for the router to select experts and it is difficult to train a MoE model with strong differentiation capabilities.

[0010] There is no explicit migration of the model’s English capabilities to low-resource capabilities: When expanding training, MoE-LPR only uses a large amount of monolingual knowledge to expand multilingual capabilities, and does not consider migrating the model’s stronger multilingual capabilities to low-resource languages.

[0011] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0012] Purpose of the invention: The technical problem to be solved by the present invention is to provide a large-model multi-language enhanced training method that can sustainably expand experts in view of the deficiencies in the prior art.

[0013] In order to solve the above technical problems, the present invention discloses a large-model multi-language enhanced training method for sustainable expansion of experts, comprising the following steps:

[0014] Step 1: extract and split the original FFN module in the large language model LLM to be trained into several new fine-grained FFN modules as fine-grained experts;

[0015] Step 2, setting a fine-grained router, selecting a preset number of newly added fine-grained experts according to a preset rule, and obtaining the output of the fine-grained router;

[0016] Step 3, setting a parent router, controlling the weight between the output of the original FFN module and the output of the fine-grained router, and calculating the final output;

[0017] Step 4, freeze the original FFN module and train the fine-grained expert separately;

[0018] Step 5, freeze the original FFN module, add the newly added fine-grained expert and perform training;

[0019] Step 6, freeze all other parameters and train only the parent router;

[0020] Step 7: Repeat steps 4 to 6 continuously to achieve large-model multi-language enhanced training of experts that can be continuously expanded.

[0021] Further, the step of obtaining the output of the fine-grained router in step 2 includes:

[0022] Step 2-1, using the fine-grained router, calculate the gating score G Finegrained (x) as follows:

[0023] G Finegrained (x) = Softmax(x·W router-Finegrained )

[0024] Where x represents the input, W router-Finegrained represents a fine-grained router, Softmax represents the Softmax function;

[0025] Step 2-2, select a preset number of new fine-grained experts, as follows:

[0026]

[0027] in, Indicates the set of top K experts selected for input x, Gfinegrainedi (x) represents the gating score of the i-th fine-grained FFN module in the current input x, and Topk represents the Topk strategy.

[0028] Step 2-3, calculating the output y of the fine-grained router, is as follows:

[0029]

[0030] Among them, E i (x) represents the i-th fine-grained expert, G j (x) represents the gating score of the jth fine-grained FFN module in the current input x.

[0031] Furthermore, the final output of the calculation described in step 3 includes:

[0032] Step 3-1, calculate the weight G between the output of the original FFN module and the output of the fine-grained router Fathe (x) as follows:

[0033] G Fathe (x) = Softmax(x·W Fahther-route )

[0034] Among them, W Fahther-route Represents the parent router;

[0035] Step 3-2, calculate the final output y′, as follows:

[0036] y′=∑G Fathe (x)·E(x)+x

[0037] Among them, E(x) is the set of the output of the original FFN module and the output of the fine-grained router.

[0038] Furthermore, the separate training of the fine-grained expert in step 4, i.e., using the pre-collected translation data to train the fine-grained expert, includes:

[0039] Step 4-1, design the loss function L NTP (θ new expert ), as follows:

[0040]

[0041] Where D is the translation data, θ new expert For newly added fine-grained experts, represents the model prediction probability, Indicates the mth word in the nth data in the translation data, d n represents the nth data in the translation, and | represents the conditional probability;

[0042] Step 4-2, modifying the instruction format of the translation data D;

[0043] Step 4-3, using the modified translation data D, inputting it into the fine-grained router in batches for forward propagation and backward propagation;

[0044] Step 4-4, calculate the loss function L NTP (θ new expert ) and use this to update the parameters of the fine-grained expert.

[0045] Furthermore, the step of adding and training the newly added fine-grained expert in step 5 includes:

[0046] Step 5-1, freeze the original FFN module;

[0047] Step 5-2, adding the newly added fine-grained expert;

[0048] Step 5-3, design the first loss function L NTP ;

[0049] Step 5-4, design the second loss function L balance , as follows:

[0050]

[0051] Where N represents the number of fine-grained experts, f i Indicates the situation where fine-grained expert i is selected, P i Represents the mean score given to fine-grained expert i in the current batch of data;

[0052] Step 5-5, design the training objectives according to the first loss function and the second loss function, as follows:

[0053]

[0054] Among them, argmin represents the value of the independent variable that makes the function achieve the minimum value, and α represents the hyperparameter;

[0055] Step 5-6: Collect multilingual and monolingual data, input it into the model, and update it through back propagation according to the above training objectives.

[0056] Furthermore, the first loss function L is designed as described in step 5-3 NTP , as follows:

[0057]

[0058] Among them, θ newrepresents the newly added fine-grained expert, W r Represents parent routers and fine-grained routers, Expand the monolingual data of multiple languages ​​for the current batch, Indicates the fth word in the eth data of the current batch of multilingual monolingual data, d e Represents the e-th data in the single-language dataset.

[0059] Furthermore, when the fine-grained expert i described in step 5-4 is selected, the calculation method is as follows:

[0060]

[0061] Where T is the number of words in the current batch of expanded multilingual monolingual data, Represents the indicator function.

[0062] Furthermore, the mean value of the scores given to the fine-grained expert i in the current batch of data described in step 5-4 is calculated as follows:

[0063]

[0064] Among them, G i (t) represents the score of word t in selecting fine-grained expert i.

[0065] Furthermore, in step 6, only the parent router is trained, that is, all parts except the parent router are frozen, and only the parent router is trained.

[0066] Further, the step 6 of training only the parent router specifically includes:

[0067] Collect monolingual data of extended languages ​​and non-extended languages, train only the parent router, and update the parent router parameters through backpropagation through the cross-entropy loss function.

[0068] Beneficial effects:

[0069] 1. Scalability: When new experts are continuously added, the model’s cost during inference remains unchanged.

[0070] 2. Significantly reduce training overhead: The parameter size of fine-grained experts is significantly smaller than the parameter size of the FFN layer in the large language model. Therefore, as the parameters are reduced, the training overhead will also be reduced.

[0071] 3. Significantly avoid the effect of catastrophic forgetting: Since there is only one router in the router strategy of MoE-LPR, the expression ability is weak. However, due to the existence of hierarchical routers, the cooperation between multiple routers can greatly alleviate the catastrophic forgetting phenomenon.

[0072] 4. Improve training stability and help models quickly migrate to multiple languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0074] Figure 1 It is a schematic diagram of the two-stage training method of MoE-LPR.

[0075] Figure 2 Schematic diagram of the three-stage training method proposed by the present invention. DETAILED DESCRIPTION

[0076] Based on the extended expert strategy of MoE, the present invention proposes a fine-grained extension scheme based on a hybrid expert model, which adds additional gating units to the fine-grained experts and the original model FFN module, dynamically controls the output weights of different modules, and retains the model's ability to avoid catastrophic forgetting while effectively expanding multiple languages.

[0077] The general idea of ​​the present invention is as follows:

[0078] Three-stage expansion of multilingual MoE: In view of the problems of unstable training, lack of differentiation of experts, and forgetting of the original language in the existing technology of training MoE, this paper proposes a three-stage training method. In the first stage, the translation data is summarized and used to train experts in parallel, which helps fine-grained experts to show basic differentiation while maintaining training stability and helping the model to transfer knowledge from English to low-resource languages. In the second stage, a large number of extended language monolinguals are used to improve the multilingual ability of the model. In the third stage, the router is trained to complete the collaborative cooperation, which helps the model to face experts with different choices in different languages.

[0079] The proposal of multi-language hierarchical routing: The present invention proposes two routings to cooperate to complete multi-language expansion. The fine-grained router selects fine-grained expansion language experts, and the parent router selects between the original language and the expansion language. This method can not only better complete the expansion of multi-languages, but also better retain the original language of the large model.

[0080] The technical aspects of the present invention are as follows:

[0081] This technical solution aims at the challenges of large language models (LLMs) in multilingual learning, and effectively expands the multilingual capabilities of existing basic large models by proposing a sustainably scalable multi-round three-stage training method. The present invention utilizes a mixed expert model (MoE) structure to achieve efficient learning of multilingual data and effectively prevent catastrophic forgetting. In each round of extended learning, the first stage is the multilingual expert hot start stage, the second stage is the multilingual capability expansion stage, and the third stage is the original language capability "review" stage. Through these measures, this solution aims to improve the sustainable scalability of LLMs in low-resource language processing and maintain knowledge of existing languages.

[0082] The basic framework of the present invention is as follows: Figure 2 As shown, the details are as follows:

[0083] The present invention is different from the general MoE loop replication stage. When a large language model LLM is given, the present invention extracts and splits the original FFN module of LLM, and the separated fine-grained FFN constitutes multiple fine-grained experts. Its scale is significantly smaller than the FFN module of the initial language model to control the consumption of computing resources. At the same time, the model controls the weight between the initial language FFN output and the fine-grained expert output by setting two routers (father router and fine-grained router Top-2router), thereby optimizing the efficiency of multiple expert selection. The specific formula is as follows:

[0084] G Finegrained (x) = Softmax(x·Wr router-Finegrained )

[0085]

[0086] Where x is the input representation of the current layer, and E is the fine-grained expert of the current layer. Among the fine-grained experts, the fine-grained router W router-Finegrain According to x, a gating score G will be obtained. Finegrained (x) represents the score of the current representation x for each fine-grained expert. According to the Topk strategy, x will select two fine-grained experts, and normalize the weights and weight them to obtain the residual output y of the fine-grained expert. Unlike MoE-LPR, which only uses one router to control the output, after obtaining the output of the fine-grained expert, the present invention will dynamically allocate the weights of the FFN output of the original language model and the output of the fine-grained expert, and weighted average to form the output of the current layer of the final model. The specific formula is as follows:

[0087] G Father (x) = Softmax(x·W Fahther-rou )

[0088] y=∑G Father (x)·E(x)+x

[0089] Where E is the set of FFN outputs of the original language model and the fine-grained expert outputs, and W Fahther-rout This structural design not only meets the demand for computing resources, but also improves the model's ability to adapt to new languages ​​and retain the original language to the greatest extent possible.

[0090] The training method of the present invention:

[0091] Phase 1: Multilingual Experts Hot Start Phase

[0092] During the training process at this stage, a simple hot start is mainly completed in advance for the upcoming MoE pre-training stage. In this stage, the present invention first extracts the FFN network of the original model and splits it into fine-grained experts. The parameters of the original model are frozen to maintain the model's ability in the mastered language, thereby preventing catastrophic forgetting. In this stage, since the newly added experts are fine-grained experts, after simple weight averaging, the output will be quite different from the original model output, which will result in a larger loss function during training. Based on this, the present invention first uses a small amount of translation data to train the newly added module. The use of translation data can first significantly reduce the order of magnitude of the loss function, so that the model is in a stable state in the subsequent stage of training. At the same time, the translation data can also help the model better transfer English knowledge to the extended language. The loss function is as follows:

[0093]

[0094] Where D is the translation data, θ new expert For newly added fine-grained experts, represents the model prediction probability, Indicates the mth word in the nth data in the translation data, d n Indicates the nth data in the translation, and | indicates the conditional probability. During the training process, the instruction format of the parallel corpus data is first modified, and then forward propagation and back propagation are performed in batches, the loss function is calculated and the parameters are updated accordingly, completing the first stage of training.

[0095] Phase 2: Multilingual Expert Expansion Phase

[0096] During the training process at this stage, MoE continues pre-training. In this stage, the parameters of the original model are frozen to maintain the model's ability in the language it has mastered, thereby preventing catastrophic forgetting. At the same time, the present invention will train the added fine-grained expert module that has been hot-started to adapt to the new language. The parameters of the new expert will be copied from the old expert, so that the knowledge of the new language can be freely selected between the old expert and the new expert. The loss function is as follows:

[0097]

[0098] Among them, θ new represents the newly added fine-grained expert, W r Represents parent routers and fine-grained routers, Expand the monolingual data of multiple languages ​​for the current batch, Indicates the fth word in the eth data of the current batch of multilingual monolingual data, d e represents the e-th data in the monolingual dataset. N represents the number of experts, K represents the value of topK, and T represents the number of words in the current batch of monolingual data expanded to multiple languages. represents the indicator function. i (t) represents the score of word t in selecting fine-grained expert i.

[0099] Based on the above formula, the optimization objectives of the second stage are as follows:

[0100]

[0101] Among them, argmin represents the value of the independent variable that makes the function achieve the minimum value, and α represents the hyperparameter;

[0102] In this stage, the present invention prepares a large amount of multilingual and monolingual data using the optimization target optimization model. The present invention updates the expert and router parameters of the model through back propagation to help the large model master the extended language knowledge.

[0103] Phase 3: Original language ability "review" phase

[0104] The "review" training in the three-stage phase focuses on restoring the ability of the original language. In this stage, only the parent router is trained, and only a small amount of data is needed to effectively restore the original language ability. In the MOE-LPR technology, a training process for restoring the original language ability is also required. The difference is that MOE-LPR sets an LPR loss in the training so that the token of the original language ability selects the FFN of the original model, and the token of the new extended language selects any FFN. The present invention splits the selection of the extended language expert and the selection of the original language and the extended language into two different functions, and implements them through different routers respectively.

[0105] The purpose of the fine-grained router is to select two of the multiple fine-grained experts as the output of the current token. The purpose of the parent router is to better balance the output of the dense original model and the output of the fine-grained expert.

[0106] In this stage, the present invention only trains the parent router, so as to achieve the fine-grained expert selection mode learned in the first stage for the tokens of the extended language, while the original language uses the parent router to keep most of the tokens more inclined to the dense model. This process effectively ensures the compatibility and effectiveness of new and old knowledge, and solves the problem of balancing multiple loss functions by adjusting hyperparameters.

[0107] Expected results of multiple rounds of training

[0108] By repeating the above three-stage process in multiple rounds of training, this technical solution aims to achieve continuous optimization and expansion of the model in multilingual learning. The addition of each round of new language data will enable the model to achieve higher transfer capabilities in the learning of the extended language and continue to maintain high performance on the original language. This sustainable learning design not only improves the flexibility and adaptability of LLM in the face of different language challenges, but also reduces the negative impact that may be caused by language distribution shift.

[0109] Example:

[0110] After three stages of training using the Qwen2.5 3b model to expand 12 fine-grained experts, compared to moelpr, the present invention significantly retains the ability to follow instructions in the original English language. As shown below, when the model is required to complete an email writing requirement, the reply of the present invention is more reasonable and useful, indicating that the model has a strong anti-forgetting ability.

[0111] Model Input: "Draft a professional email seeking your supervisor's feedback on the 'Quarterly Financial Report' you prepared. Ask specifically about the data analysis, presentation style, and the clarity of conclusions drawn. Keep the email short and to the point.", "Take a moment to evaluate and critique your own response."

[0112] Answer of the present invention:

[0113] Request for Feedback on Quarterly Financial Report Dear [Supervisor's Name], I hope this email finds you well. I have prepared the quarterly financial report and would like your feedback on the data analysis, presentation style, and the clarity of conclusions drawn. I would appreciate it if you could take a few minutes to review the report and provide your insights. Thank you for your time and consideration. Best regards, [Your Name] [Your Position]"

[0114] Answer of Moelpr:

[0115] "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nDraft a professional email seeking your supervisor's feedback on the'Quarterly Financial Report'you prepared. Ask specifically about the data analysis, presentation style, and the clarity of conclusions drawn. Keep the email short and to the point.<|im_end|>\n<|im_start|>assistant\n","<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nDraft aprofessional email seeking your supervisor's feedback on the'QuarterlyFinancial Report'you prepared.Ask specifically about the data analysis,presentation style,and the clarity of conclusions drawn.Keep the email short and to the point.<|im_end|>\n<|im_start|>assistant\n|||first turn answer|||<|im_end|>\n<|im_start|>user\nTake a moment to evaluate and critique your ownresponse.<|im_end|>\n<|im_start|>assistant\n

[0116] It can be seen that the answer given by the present invention is more reasonable.

[0117] In terms of extended language, the score of this invention is as high as 77 points on the public multilingual evaluation reading comprehension dataset, which greatly improves the multilingual ability of the model.

[0118] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a large-model multi-language enhanced training method for sustainable expansion of experts provided by the present invention and some or all of the steps in each embodiment can be run. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0119] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0120] The present invention provides a method and idea for a large-scale multi-language enhanced training method for sustainable expansion of experts. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A large-model multi-language enhanced training method that can sustainably expand experts, characterized in that: The following steps are involved: Step 1: extract and split the original FFN module in the large language model LLM to be trained into several new fine-grained FFN modules as fine-grained experts; Step 2, setting a fine-grained router, selecting a preset number of newly added fine-grained experts according to a preset rule, and obtaining the output of the fine-grained router; Step 3, setting a parent router, controlling the weight between the output of the original FFN module and the output of the fine-grained router, and calculating the final output; Step 4, freeze the original FFN module and train the fine-grained expert separately; Step 5, freeze the original FFN module, add the newly added fine-grained expert and perform training; Step 6, freeze all other parameters and train only the parent router; Step 7: Repeat steps 4 to 6 continuously to achieve large-model multi-language enhanced training of experts that can be continuously expanded.

2. The large-model multi-language enhanced training method for sustainable expansion of experts according to claim 1 is characterized in that: The step 2 of obtaining the output of the fine-grained router includes: Step 2-1, using the fine-grained router, calculate the gating score G Finegrained (x) as follows: G Finegrained (x)=Softmax(x·W router-Fineg ) Where x represents the input, W router-Finegrained represents a fine-grained router, Sofmtax represents a Sofmtax function; Step 2-2, select a preset number of new fine-grained experts, as follows: in, Indicates that the input x selects the set of top K experts, represents the gating score of the i-th fine-grained FFN module in the current input x, and Topk represents the Topk strategy; Step 2-3, calculating the output y of the fine-grained router, is as follows: Among them, E i (x) represents the i-th fine-grained expert, G j (x) represents the gating score of the jth fine-grained FFN module in the current input x.

3. The large-model multi-language enhanced training method for sustainable expansion of experts according to claim 2 is characterized in that: The final output of the calculation described in step 3 includes: Step 3-1, calculate the weight G between the output of the original FFN module and the output of the fine-grained router Fathe (x) as follows: G Fathe (x)=Softmax(x·W Fahther-rou ) Among them, W Fahther-route Represents the parent router; Step 3-2, calculate the final output y′, as follows: y′=ΣG Fathe (x)·E(x)+x Among them, E(x) is the set of the output of the original FFN module and the output of the fine-grained router.

4. The large-model multi-language enhanced training method for sustainable expansion of experts according to claim 3 is characterized in that: The separate training of the fine-grained expert described in step 4, i.e., using the pre-collected translation data to train the fine-grained expert, includes: Step 4-1, design the loss function L NTP (θ newexpert ), as follows: Where D is the translation data, θ newexpert For newly added fine-grained experts, represents the model prediction probability, Indicates the mth word in the nth data in the translation data, d n represents the nth data in the translation, and | represents the conditional probability; Step 4-2, modifying the instruction format of the translation data D; Step 4-3, using the modified translation data D, inputting it into the fine-grained router in batches for forward propagation and backward propagation; Step 4-4, calculate the loss function L NTP (θ newexpert ) and use this to update the parameters of the fine-grained expert.

5. The large-model multi-language enhanced training method for sustainable expansion of experts according to claim 4 is characterized in that: The step 5 of adding and training the newly added fine-grained expert includes: Step 5-1, freeze the original FFN module; Step 5-2, adding the newly added fine-grained expert; Step 5-3, design the first loss function L NTP ; Step 5-4, design the second loss function L balance , as follows: Where N represents the number of fine-grained experts, f i Indicates the situation where fine-grained expert i is selected, P i Represents the mean score given to fine-grained expert i in the current batch of data; Step 5-5, design the training objectives according to the first loss function and the second loss function, as follows: Among them, argmin represents the value of the independent variable that makes the function achieve the minimum value, and α represents the hyperparameter; Step 5-6: Collect multilingual and monolingual data, input it into the model, and update it through back propagation according to the above training objectives.

6. The large-model multi-language enhanced training method for sustainable expansion of experts according to claim 5 is characterized in that: Design the first loss function L as described in step 5-3 NTP , as follows: Among them, θ new represents the newly added fine-grained expert, W r Represents parent routers and fine-grained routers, Expand the monolingual data of multiple languages ​​for the current batch, Indicates the fth word in the eth data of the current batch of multilingual monolingual data, d e Represents the e-th data in the single-language dataset.

7. The large-model multi-language enhanced training method for sustainable expansion of experts according to claim 6 is characterized in that: When the fine-grained expert i described in step 5-4 is selected, the calculation method is as follows: Where T is the number of words in the current batch of expanded multilingual monolingual data, Represents the indicator function.

8. The large-model multi-language enhanced training method for sustainable expansion of experts according to claim 7, characterized in that: The mean score given to fine-grained expert i in the current batch of data described in step 5-4 is calculated as follows: Among them, G i (t) represents the score of word t in selecting fine-grained expert i.

9. The large-model multi-language enhanced training method for sustainable expansion of experts according to claim 8, characterized in that: The step 6 of training only the parent router means freezing all parts except the parent router and only training the parent router.

10. The large-model multi-language enhanced training method for sustainable expansion of experts according to claim 9, characterized in that: Train only the parent router as described in step 6, specifically: Collect monolingual data of extended languages ​​and non-extended languages, train only the parent router, and update the parent router parameters through backpropagation through the cross-entropy loss function.

Images