Large model multi-language enhancement model fusion method based on mixed experts

Abstract

The invention provides a large-model multi-language enhancement model fusion method based on mixed experts, which comprises the following steps of: 1, selecting a to-be-fused layer from an anchor point model and a language expert model; step 2, constructing a fusion module; 3, fusing each to-be-fused layer in the anchor point model and the language expert model by using a fusion module to obtain a fused model; and step 4, training the fused model to obtain a final fusion result, and completing large-model multi-language enhancement model fusion based on the hybrid experts.

Description

Technical Field

[0001] The present invention relates to a large-model multi-language enhancement model fusion method, in particular to a large-model multi-language enhancement model fusion method based on mixed experts. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] In the field of natural language processing (NLP), with the improvement of computing power and the accumulation of massive data, a variety of advanced technical solutions have emerged. Among them, large language models (LLMs) have become a hot topic of research due to their powerful text understanding and generation capabilities. These models, such as GPT-4 (reference: Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report [J]. arXiv preprint arXiv: 2303.08774, 2023.) and Qwen (reference: Bai J, Bai S, Chu Y, et al. Qwen technical report [J]. arXiv preprint arXiv: 2309.16609, 2023.), can perform well in many NLP tasks, including text generation, text summarization, machine translation, and sentiment analysis, after large-scale unsupervised pre-training. At the same time, these models have shown significant advantages in processing grammatical structures, contextual understanding, and word sense disambiguation of specific languages ​​(especially resource-rich languages ​​such as English). However, most existing LLMs are optimized for a specific language, which significantly reduces their performance when processing non-mainstream languages. This tendency to optimize for a single language leads to their limitations in multilingual environments. Although they have strong single-language capabilities, they lack the ability to effectively handle multiple languages, which seriously limits their widespread application in a global environment and creates certain obstacles for users who want to communicate and obtain information across cultures.

[0004] MoE (Mixture of Experts) is a deep learning model architecture that selects different experts (sub-models) to process input data through a dynamic routing mechanism. The core idea of ​​MoE is to activate and train only some experts, rather than all experts, when given an input, so as to reduce computational overhead while ensuring a large model size. Currently, MoE technology is usually used in the Transformer model, taking the forward feedback module of each layer as its expert, and combining it with its unique routing mechanism to form a MoE module.

[0005] Model fusion refers to the fusion of two models with different capabilities to obtain a fusion model that can handle multiple tasks. The fusion model generally exceeds the performance of its sub-models in the tasks that each sub-model is good at. Nowadays, it is very expensive to train a large model from scratch, and there are many models focusing on different fields. Model fusion technology hopes to integrate these models and obtain a stronger model at a low cost. The current fusion solutions mainly include fusion at the output level and fusion at the parameter level.

[0006] The commonly used fusion scheme in the prior art is CALM (reference: Bansal R, Samanta B, Dalmia S, et al. LLM Augmented LLMs: Expanding Capabilities through Composition [C] / / The Twelfth International Conference on Learning Representations.). The CALM framework adopts a parameter-based model fusion method, aiming to effectively combine different language models to enhance their capabilities in multilingual processing tasks. CALM fuses homologous models, where the smaller model continues to pre-train on low-resource languages ​​to provide knowledge of low-resource languages, and the larger model provides powerful reasoning capabilities, so that the fused model shows better performance in processing resource-scarce languages.

[0007] Although CALM successfully integrates homologous models, it still has some shortcomings. For example, CALM integrates the intermediate representation layer of the sub-model through cross-attention, and then directly adds the result of the integration calculation to the intermediate representation of the anchor model. This method lacks an effective gating mechanism, which makes it difficult for the integration model to flexibly control the weight of the integration knowledge and the anchor knowledge. For example, the shallow layer of the large model requires more grammatical and semantic knowledge of the language, while the intermediate layer requires more reasoning ability. The lack of an effective and flexible gating mechanism will cause the model to inject unnecessary redundant information into the integration layer, interfering with the generation of the model. In addition, each layer of the Transformer contains a Self-attention module and an MLP module. CALM only integrates the output representation of the MLP, and does not integrate the representation and output of the Self-attention. For low-resource languages, the model's attention learning of its sentences is crucial. Therefore, the lack of the attention information fusion process between the language expert model and the anchor model may lead to insufficient ability fusion. In summary, the shortcomings of the existing technology mainly include the lack of a flexible and effective gating mechanism and the lack of fusion between the attention knowledge of the sub-models.

[0008] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0009] Purpose of the invention: The technical problem to be solved by the present invention is to provide a large-model multi-language enhancement model fusion method based on mixed experts in view of the shortcomings of the prior art.

[0010] In order to solve the above technical problems, the present invention discloses a large model multi-language enhancement model fusion method based on mixed experts, comprising the following steps:

[0011] Step 1: Select the layer to be fused from the anchor model and the language expert model;

[0012] Step 2, construct the fusion module;

[0013] Step 3: Use the fusion module to fuse each layer to be fused in the anchor model and the language expert model to obtain a fused model;

[0014] Step 4: train the fused model to obtain the final fusion result, thereby completing the large-model multi-language enhanced model fusion based on mixed experts.

[0015] Furthermore, the step of selecting the layer to be fused in step 1 includes:

[0016] Assume that the anchor model and the language expert model have m layers and n layers respectively, where m>n; select k layers from each of them for fusion, where 1≤k≤n;

[0017] The layers are selected in such a way that the spacing between two adjacent layers is constant.

[0018] Furthermore, the step 1 of ensuring that the distance between two adjacent layers is constant includes:

[0019] The sequence numbers of the layers to be fused in the anchor model and the language expert model are arithmetic progressions:

[0020] (l2-l1)=…=(l k -l k-1 )

[0021] Among them, l k Indicates the sequence number of the k-th layer to be fused.

[0022] Furthermore, the step 3 described in step 3 is to fuse each layer to be fused in the anchor model and the language expert model, including:

[0023] Assume that the i-th layer of the language expert model is fused with the j-th layer of the anchor point model, that is, the self-attention module and the feedforward neural network module in the i-th layer of the language expert model and the j-th layer of the anchor point model are fused respectively.

[0024] Furthermore, the fusion of each to-be-fused layer in the anchor model and the language expert model described in step 3 specifically includes:

[0025] Step 3-1, performing fusion calculation on the output representation of the attention module of the i-th layer of the language expert model and the output representation of the attention module of the j-th layer of the anchor model through the fusion module, and using the output of the fusion calculation as the input of the feedforward neural network module of the j-th layer of the anchor model;

[0026] Step 3-2, the output representation of the feedforward neural network module of the i-th layer of the language expert model and the output representation of the feedforward neural network module of the j-th layer of the anchor model are fused through the fusion module, and the output is used as the input of the j+1-th layer of the anchor model.

[0027] Furthermore, the construction of the fusion module described in step 2 includes:

[0028] Step 2-1, construct a mapping layer, and transform the layer representation vector H of the language expert model 语言专家 , mapped to the same dimension as the anchor model, as follows:

[0029] H s =H 语言专家 ×W 映射层

[0030] Among them, H s represents the layer representation vector of the language expert model after dimension conversion, W 映射层 represents the dimension conversion parameter matrix;

[0031] Step 2-2, construct the linear mapping module W Q , W K and W V ;

[0032] Wherein, the layer representation vector H of the anchor model is 锚点模型 , after the linear mapping module W Q After calculation, the query vector used as the cross-attention module is as follows:

[0033] Query = H 锚点模型 ×W Q

[0034] The layer representation vector H of the language expert model after the dimension conversions , after the linear mapping module W K After calculation, the Key vector used as the cross-attention module is as follows:

[0035] Key=H S ×W K

[0036] The layer representation vector H of the language expert model after the dimension conversion s , after the linear mapping module W V After calculation, the Value vector used as the cross-attention module is as follows:

[0037] Value=H S ×W V

[0038] Step 2-3, construct a cross attention module, input the Query vector, Key vector and Value vector into the cross attention module for fusion, and obtain the fused knowledge representation H CA , as follows:

[0039] H CA =DrossAttention(Query,Key,Value))

[0040] Among them, CrossAttention means cross attention fusion;

[0041] Step 2-4, build a router and obtain the output H of the fusion module merge .

[0042] Further, the output H of the fusion module is obtained in step 2-4 merge ,include:

[0043] Step 2-4-1, using a binary classifier as a router to represent H CA As input, the output dimension is 2 probability distribution, and the probability distribution is used as the layer representation vector H of the anchor model. 锚点模型 and fusion knowledge representation H CA The weight G1(H CA ) and G2(H CA ), as follows:

[0044] G(H CA )=Softmax(H CA ×W 路由器 )

[0045] Among them, Softmax represents the normalization calculation, W 路由器represents the parameter matrix of the router, G(H CA ) represents a probability distribution containing two elements, G1(H CA ) and G2(H CA ) represent the values ​​of two elements respectively;

[0046] Step 2-4-2, calculate the weighted sum H of the two merge , as follows:

[0047] H merge =G1(H CA )×H 锚点模型 +G2(H CA )×H CA

[0048] Among them, H merge That is the output of the fusion module.

[0049] Furthermore, the training fused model described in step 4 includes:

[0050] Step 4-1, construct the divergence loss function L 分歧 (H CA ,H 锚点模型 ), as follows:

[0051]

[0052] Among them, α is the loss coefficient hyperparameter, i represents the subscript of the vector element, and the present invention adds divergence loss to all fusion layers for optimization;

[0053] Step 4-2, using multilingual data as a training set, inputting the fused model for training, freezing all other parameters of the anchor model and the language expert model during training, and only training the parameters of the linear mapping module, the cross attention module and the router.

[0054] Furthermore, the anchor model described in step 1 has a model size > 7B.

[0055] Furthermore, the language expert model described in step 1 has a model size <1B.

[0056] Beneficial effects:

[0057] 1. The present invention introduces a gating mechanism based on hybrid expert technology to effectively control the information flow of sub-models, intelligently adjust the weight of sub-model knowledge at layers with different requirements for sub-model knowledge, reduce the redundant information that may be brought by cross-attention modules, and utilize the knowledge of each sub-model more efficiently.

[0058] 2. In a multilingual scenario, the attention module of the language expert model contains a wealth of low-resource language grammatical semantic knowledge. The CALM fusion solution is difficult to utilize this knowledge, resulting in insufficient fusion. The present invention fuses the attention matrices of the sub-models through hybrid expert technology, inserts this knowledge into the fusion process, promotes the deep fusion of the sub-models, and enhances the multilingual capabilities of the fusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0060] Figure 1 Schematic diagram of the CALM model fusion method.

[0061] Figure 2 This is a schematic diagram of the overall structure of the model fusion method proposed in this invention.

[0062] Figure 3 It is a structural diagram of the fusion module. DETAILED DESCRIPTION

[0063] Based on the CALM fusion scheme, this paper proposes a deep fusion scheme based on a hybrid expert model, which aims to add a flexible and effective gating unit to the fusion mechanism and further proposes a self-attention representation within the fusion layer to reduce the granularity of model fusion. This paper further enhances the low-resource language capabilities of the fusion model.

[0064] The general idea of ​​the present invention is as follows:

[0065] 1. Introduction of gating unit: Based on the cross-attention mechanism, the present invention introduces the gating unit to achieve weighted fusion of sub-model knowledge, endow the fusion model with more flexible and efficient knowledge flow control, make full use of the strengths of the sub-models, and enhance the multilingual ability of the model.

[0066] 2. Fusion of self-attention module output: This invention newly introduces the fusion of self-attention module output. In a multilingual scenario, attention learning of low-resource languages ​​is also crucial. Further fusing this part of information is conducive to promoting deeper information interaction among sub-models and improving the upper limit of the fusion model.

[0067] The specific technical solutions of the present invention are as follows:

[0068] The present invention proposes MOEA (Mixture of Experts and Attentions, a combined enhanced language model framework based on mixed experts), which aims to effectively and practically combine the existing basic large model with the homologous specific language expert model to further enhance the degree of fusion between models and improve the multilingual ability of the fusion model.

[0069] MOEA follows the framework of CALM. Given a large language model Large-LLM (anchor model, which is a large model with strong English or Chinese capabilities, model size>7B) and an enhanced small model Small-LLM (language expert model, which is a small language expert model that has been pre-trained in multiple languages, model size<1B), MoEA aims to deeply combine these two models to obtain a model with stronger comprehensive capabilities. For example, the anchor model performs better in English reasoning tasks, while the language expert model has a better understanding of the semantics and grammar of low-resource languages. The present invention hopes to use a smaller training cost without changing the weights of the two models, and mix the advantages of the two models to obtain a combined model that is better at mathematical reasoning in low-resource languages.

[0070] Among them, CALM introduces a cross-model attention mechanism. Specifically, Figure 1 As shown in Figure 1, CALM uses a larger model with stronger reasoning ability as the anchor model, and uses a smaller model with the same origin as the larger model for specific task training enhancement to obtain a customized smaller model. Then, the outputs of the customized smaller model and the anchor model at certain fixed layers are calculated using a cross-model cross-attention mechanism, and the middle layer representation of the anchor model is used as the query vector of the cross-attention (W Q ), the middle layer representation of the customized small model is first mapped to the same dimension as the anchor model through a linear projection layer, and then the result is used as the key of the cross attention (W K ) and Value(W V ) vector, the calculation result of the cross attention is directly added to the intermediate representation of the anchor model as the input of the next layer of the anchor model. CALM works well on multilingual tasks. It uses the pre-trained small model of the low-resource language as a customized small model, uses the combined data of the low-resource language and the high-resource language, freezes the parameters of the anchor model and the small model customized by the language expert, and only trains the cross attention module to ensure that the capabilities of the sub-model are not forgotten, while integrating the capabilities of the sub-model.

[0071] The present invention regards the cross attention module used by CALM as a knowledge fusion expert, and the inter-layer representation of the anchor model as a reasoning expert, and then adds a linear gating unit to form a hybrid expert module (MoE) with it, and flexibly controls the fusion ratio of fusion knowledge and reasoning ability through the gating unit. In addition, MOEA also adds the fusion calculation of the output representation of the self-attention module. The fusion strategy of this part is consistent with the above-mentioned hybrid expert module and becomes a hybrid attention module (MoA).

[0072] The core module of the present invention is a hybrid expert fusion strategy, which is not only the fusion of the output of the layer, but also the fusion of the attention and feedforward modules within the layer, such as Figure 2 and Figure 3 As shown in the figure, they are the fine-grained fusion framework and the fusion module, which are as follows:

[0073] Fine-grained fusion framework:

[0074] Assume that the anchor model and the language expert model have m and n layers respectively, and select k layers from them for fusion (1≤k≤n). The layers are selected in such a way as to ensure that the spacing between two adjacent layers is constant, that is, the subscripts (serial numbers) of the layers that the anchor model and the language expert model need to fuse are arithmetic progressions:

[0075] (l2-l1)=…=(l k -l k-1 )

[0076] Where l represents the subscript of the model layer, such as Figure 2 As shown, the i-th layer of the language expert model is fused with the j-th layer of the anchor model. Each layer of the Transformer includes a self-attention module and a feedforward neural network module, and the present invention deeply fuses the two modules respectively. The output representation of the attention module of the i-th layer of the language expert model and the output representation of the attention module of the j-th layer of the anchor model are fused through a fusion module, and the output is used as the input of the feedforward network of the j-th layer of the anchor model. Then, the output representation of the feedforward network of the i-th layer of the language expert model and the output representation of the feedforward network of the j-th layer of the anchor model are fused through a fusion module, and the output is used as the input of the j+1-th layer of the anchor model for subsequent calculations.

[0077] Fusion module design

[0078] like Figure 3 As shown in the figure, the fusion module is responsible for fusing the internal representations from the two models through cross attention and routing. Since the representation dimension of the language expert model is smaller than that of the anchor model, the fusion module first maps the representation of the language expert model to the dimension of the anchor model:

[0079] H s =H语言专家 ×W 映射层

[0080] Among them, H s represents the intermediate representation vector of the language expert model after dimension conversion, H 语言专家 Represents the intermediate representation vector of the language expert model, W 映射层 represents the dimension conversion parameter matrix; H 锚点模型 is the layer representation vector of the anchor model. Then H 锚点模型 After the linear mapping module W Q Calculate the Query vector as a cross-attention module, H s After the linear mapping module W k Get the Key vector, and pass it through the linear mapping module W V The Value vector is obtained, and finally the fusion representation is obtained through cross-attention calculation:

[0081] Query = H 锚点模型 ×W Q

[0082] Key=H S ×W K

[0083] Value=H S ×W V

[0084] H CA =CrossAttention(Query,Key,Value))

[0085] After cross-attention fusion, the low-resource language knowledge carried by the language expert model and the reasoning knowledge carried by the anchor model exist in the result vector, which is called the fused knowledge representation H CA .

[0086] like Figure 2 As shown in FIG. 1 , in order to better control the influence of fusion knowledge on the anchor model, the present invention adds a router. The router is a binary classifier that uses fusion knowledge to represent H CA As input, the output dimension is a probability distribution of 2, and the probability distribution is used as the weight of the anchor model representation and the fusion knowledge representation, and the weighted sum of the two is calculated:

[0087] G(H CA )=Softmax(H CA ×W 路由器 )

[0088] H merge =G1(H CA )×H 锚点模型 +G2(HCA )×H CA

[0089] Among them, W 路由器 Represents the parameter matrix of the router. After the linear layer maps the fusion representation to a dimension of 2, it performs Softmax normalization calculation and uses the obtained results as the weights of the fusion representation and the anchor model representation respectively. The weighted sum of the two is used as the input of the next module of the anchor model. This gating unit enables the fusion model to learn how to adjust the combined weights of fusion knowledge and reasoning ability to achieve a more efficient and sufficient knowledge ratio, making the model more fully integrated, further stimulating the potential of the fusion solution, and improving the multilingual capabilities of the fusion model.

[0090] Training method:

[0091] In order to prevent H CA With H L In order to achieve better expression results by tending to homogeneity during training, the present invention imposes divergence loss on the two representations and imposes penalties on representations that tend to be homogeneous. Specifically, the present invention calculates the KL divergence loss of the two representations and adjusts the degree of penalty through a controllable loss coefficient hyperparameter α. The goal of the divergence loss is to minimize the KL divergence of the distribution of the two representations:

[0092]

[0093] MOEA is trained on high-quality multilingual data, during which the parameters of the anchor model and the language expert model will be completely frozen, and only the parameters of the above-mentioned linear mapping module, cross-attention module, and gating module will be trained. MOEA integrates homologous models, that is, a series of models with the same architecture and vocabulary, such as the Qwen2 series, and selects the smallest model for pre-training of low-resource language knowledge, storing a large amount of grammatical and semantic knowledge of low-resource languages. This model is used as a language expert model, and a larger model is selected as an anchor model to provide powerful reasoning capabilities. After MOEA fuses the sub-models, the fused model will have better low-resource language capabilities.

[0094] Example:

[0095] Qwen2.5-Math-7B is used as the anchor model, and the Qwen-2.5-0.5B model is further pre-trained using a large amount of multilingual data to obtain a language expert model. Then, the anchor model and the language expert model are fused and trained by MOEA to obtain a fusion model. As shown below, when the model is asked to solve a Bengali math problem, the model of the present invention responds with the correct reasoning process and results, indicating that the model is more fully integrated.

[0096] Model input:

[0097] (Explanation: The product of the three numbers is 50. But the product of x and y is 5. What is z?)

[0098] MOEA model answers (Chinese explanation in brackets):

[0099] ●

[0100] ● (Now we need to calculate the value of z)

[0101] ● (We can calculate the value of z by dividing the product of x×y×z by the product of x×y.

[0102] ●

[0103] ●

[0104] ●

[0105] (So ​​z = 10)

[0106] The CALM model answered incorrectly and contained a lot of garbled content.

[0107] In a public multi-language math test, the fusion model of the present invention achieved an average accuracy of 54.8% on three low-resource languages, significantly exceeding the 20.93% of the anchor model before fusion.

[0108] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a large model multi-language enhanced model fusion method based on hybrid experts provided by the present invention and some or all of the steps in each embodiment. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0109] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0110] The present invention provides a thought and method for a large model multi-language enhancement model fusion method based on mixed experts. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A large model multi-language enhancement model fusion method based on mixed experts, characterized in that: The following steps are involved: Step 1: Select the layer to be fused from the anchor model and the language expert model; Step 2, construct the fusion module; Step 3: Use the fusion module to fuse each layer to be fused in the anchor model and the language expert model to obtain a fused model; Step 4: train the fused model to obtain the final fusion result, thereby completing the large-model multi-language enhanced model fusion based on mixed experts.

2. According to the method of claim 1, a large model multi-language enhancement model fusion method based on mixed experts is characterized in that: The step 1 of selecting the layer to be fused includes: Assume that the anchor model and the language expert model have m layers and n layers respectively, where m>n; select k layers from each of them for fusion, where 1≤k≤n; The layers are selected in such a way that the spacing between two adjacent layers is constant.

3. According to the method of large-model multi-language enhancement model fusion based on mixed experts in claim 2, it is characterized in that: The method described in step 1 to ensure that the spacing between two adjacent layers is constant includes: The sequence numbers of the layers to be fused in the anchor model and the language expert model are arithmetic progressions: (l2-l1)＝…＝(l k -l k-1 ) Among them, l k Indicates the sequence number of the k-th layer to be fused.

4. According to the method of claim 3, a large model multi-language enhancement model fusion method based on mixed experts is characterized in that: The step 3 described in step 3 is to fuse each layer to be fused in the anchor model and the language expert model, including: Assume that the i-th layer of the language expert model is fused with the j-th layer of the anchor point model, that is, the self-attention module and the feedforward neural network module in the i-th layer of the language expert model and the j-th layer of the anchor point model are fused respectively.

5. According to the method of claim 4, a large model multi-language enhancement model fusion method based on mixed experts is characterized in that: The fusion of each to-be-fused layer in the anchor model and the language expert model described in step 3 specifically includes: Step 3-1, performing fusion calculation on the output representation of the attention module of the i-th layer of the language expert model and the output representation of the attention module of the j-th layer of the anchor model through the fusion module, and using the output of the fusion calculation as the input of the feedforward neural network module of the j-th layer of the anchor model; Step 3-2, the output representation of the feedforward neural network module of the i-th layer of the language expert model and the output representation of the feedforward neural network module of the j-th layer of the anchor model are fused through the fusion module, and the output is used as the input of the j+1-th layer of the anchor model.

6. The method for fusion of large models and multi-language enhancement models based on mixed experts according to claim 5 is characterized in that: The construction fusion module described in step 2 includes: Step 2-1, construct a mapping layer, and transform the layer representation vector H of the language expert model 语言专家 , mapped to the same dimension as the anchor model, as follows: H S =H 语言专家 ×W 映射层 Among them, H S represents the layer representation vector of the language expert model after dimension conversion, W 映射层 represents the dimension conversion parameter matrix; Step 2-2, construct the linear mapping module W Q , W K and W V ; Wherein, the layer representation vector H of the anchor model is 锚点模型 , after the linear mapping module W Q After calculation, the query vector used as the cross-attention module is as follows: Query=H 锚点模型 ×W Q The layer representation vector H of the language expert model after the dimension conversion s , after the linear mapping module W K After calculation, the Key vector used as the cross-attention module is as follows: Key=H S ×W K The layer representation vector H of the language expert model after the dimension conversion s , after the linear mapping module W V After calculation, the Value vector used as the cross-attention module is as follows: Value=H S ×W V Step 2-3, construct a cross attention module, input the Query vector, Key vector and Value vector into the cross attention module for fusion, and obtain the fused knowledge representation H CA , as follows: H CA =CrossAttention(Query,Key,Value)) Among them, CrossAttention means cross attention fusion; Step 2-4, build a router and obtain the output H of the fusion module merge .

7. The method for fusion of large models and multi-language enhancement models based on mixed experts according to claim 6 is characterized in that: The output H of the fusion module is obtained in step 2-4 merge ,include: Step 2-4-1, using a binary classifier as a router to represent H CA As input, the output dimension is 2 probability distribution, and the probability distribution is used as the layer representation vector H of the anchor model. 锚点模型 and fusion knowledge representation H CA The weight G1(H CA ) and G2(H CA ), as follows: G(H CA )=Softmax(H CA ×W 路由器 ) Among them, Softmax represents the normalization calculation, W 路由器 represents the parameter matrix of the router, G(H CA ) represents a probability distribution containing two elements, G1(H CA ) and G2(H CA ) represent the values ​​of two elements respectively; Step 2-4-2, calculate the weighted sum H of the two merge , as follows: H merge =G1(H CA )×H 锚点模型 +G2(H CA )×H CA Among them, H merge That is the output of the fusion module.

8. The method for fusion of large models and multi-language enhancement models based on mixed experts according to claim 7 is characterized in that: The trained fused model described in step 4 includes: Step 4-1, construct the divergence loss function L 分歧 (H CA ,H 锚点模型 ), as follows: Among them, α is the loss coefficient hyperparameter, i represents the subscript of the vector element, and divergence loss is added to all fused layers for optimization; Step 4-2, using multilingual data as a training set, inputting the fused model for training, freezing all other parameters of the anchor model and the language expert model during training, and only training the parameters of the linear mapping module, the cross attention module and the router.

9. The method for fusion of large models and multi-language enhancement models based on mixed experts according to claim 1, characterized in that: Anchor model as described in step 1, model size > 7B.

10. The method for fusion of large models and multi-language enhancement models based on mixed experts according to claim 1, characterized in that: The language expert model described in step 1 has a model size < 1B.

Images