Method and system for cross-lingual task capability transfer and behavior preservation of low-resource multilingual large language model and electronic device

Abstract

The application relates to the field of artificial intelligence, in particular to a low-resource multilingual large language model cross-language same-task capability migration and behavior maintenance method and system and electronic equipment, aiming to solve the model adaptation problem of simultaneously having task capability complementation and behavior stable maintenance. The method comprises the following steps: freezing original parameters, routing parameters, unselected expert parameters and shared dense layer parameters of a pre-training dynamic mixed expert backbone model; screening high-conflict expert branches, calculating an effective projection matrix according to the frozen original projection weight, a shared residual branch, a language group residual branch and a task private residual branch; updating the task private residual branch by using a task private capability transmitter; performing double distillation constraint on the model; and training the model based on an output level distillation loss, a hidden state level representation maintenance loss, a subspace orthogonal regularization loss and a task supervision loss. The scheme provided by the application significantly improves the efficiency and stability of low-resource language task migration.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, system and electronic device for cross-language same-task ability transfer and behavior preservation of a low-resource multilingual large language model. Background Technology

[0002] With the widespread application of large language models in tasks such as translation, question answering, code generation, and mathematical reasoning, multilingual and multitasking capabilities have become crucial foundations for the practical application and deployment of these models. However, in real-world multilingual scenarios, the distribution of supervised data across different language and task combinations is highly uneven: high-resource languages ​​often have abundant instruction fine-tuning data or publicly labeled data, while low-resource languages, minority languages, or languages ​​with significant differences in writing systems typically lack sufficient supervised data for target language tasks. Furthermore, the continuous introduction of low-resource languages ​​alters the training distribution, potentially leading to an overall degradation of the model's original language capabilities, internal representations, and stable task execution behavior. This presents two core technical challenges: cross-language, same-task capability transfer and catastrophic forgetting suppression.

[0003] Existing multilingual model adaptation methods typically treat the problem as improving language modeling capabilities, rather than the cross-language transfer and retention of task execution behaviors. These task execution behaviors include not only generating correct answers but also following instructional constraints, maintaining a fixed output format, satisfying task-specific structures, controlling response language, and maintaining a stable answering style. For low-resource languages, simply improving target language fluency through continuous pre-training may not enable the model to learn the corresponding task's solution process and execution methods. Relying on translated or manually labeled data incurs high data construction costs and is often difficult to implement in extremely low-resource languages. Furthermore, low-resource adaptation may weaken the model's previously stable mastery of language representations, output formats, behavioral constraints, and answering patterns, causing the model to gain local gains on new language tasks while forgetting its abilities in preserving language task pairs.

[0004] Among the existing multilingual model adaptation techniques, the first type is continuous pre-training or target language adaptation. This type of method continues to train the model using monolingual corpora of the target language to improve the lexical, syntactic, and semantic modeling capabilities of the target language. Its advantage is that it does not require a large amount of task annotation, but it has obvious problems: this type of method mainly optimizes the modeling quality at the language level and cannot explicitly transfer the task-solving process, output format, and instruction compliance patterns already learned in high-resource languages. In addition, there is a problem of confusion between language adaptation and task behavior transfer. It can only improve the fluency of the target language, but cannot guarantee that the model will master task behaviors such as question answering, translation, formatted output, and instruction compliance. It lacks a mechanism to decouple task execution behavior from language factors and reuse it across languages.

[0005] The second type is instruction fine-tuning, data translation, and cross-language data augmentation methods. These methods translate task data in high-resource languages ​​into the target language or construct new instruction data for low-resource languages, and then perform supervised fine-tuning of the model. They can provide direct task supervision, but they have some prominent problems: they require a lot of translation, verification, and annotation work, and in low-resource language scenarios, it is difficult to construct supervision data for the target language task. For minority languages ​​or languages ​​with scarce digital resources, the quantity, quality, and annotation infrastructure of data are insufficient, resulting in high data construction costs and difficulty in guaranteeing quality. They are often difficult to implement in extremely low-resource languages.

[0006] The third type is the parameter-efficient fine-tuning method, which includes methods such as adapter, LoRA, and prefix fine-tuning. It achieves task or language adaptation by adding a small number of parameters, which can reduce training costs and reduce perturbation to the backbone model. However, it has obvious shortcomings: traditional parameter-efficient methods usually bind the adaptation parameters to a single language or task, and lack a mechanism to treat task behavior as an independent, composable, and cross-language transferable capability unit. At the same time, similar to full fine-tuning or adding complete experts, if a large number of parameters are added for each language or task, it will lead to a significant increase in deployment costs, storage costs, and training costs, which is not conducive to rapid expansion to more low-resource languages.

[0007] The fourth type is the hybrid expert architecture and multilingual expert partitioning method. Hybrid expert models assign different experts to different input lexical units or languages ​​through routing mechanisms, thereby improving model capacity and multilingual coverage. However, existing related methods have obvious defects: they focus more on capacity expansion, routing efficiency, or language-level expert division of labor, and have not fully solved the problem of gradient conflicts between different language task pairs in the same expert. Different tasks may be routed to the same expert and generate opposite gradient directions, causing parameter updates to cancel each other out or destroy each other, affecting the stability of low-resource transfer, and do not provide a clear training path to instantiate high-resource language task behavior into target language task pairs.

[0008] The fifth type is catastrophic forgetting mitigation methods. Existing methods usually use replay samples, regularization constraints, or knowledge distillation to maintain the original task capabilities, but the problems are not to be ignored: most methods only focus on the output distribution or parameter distance, without simultaneously constraining the model's response-level behavior and the internal representation of the hybrid expert layer. Therefore, it is difficult to stably maintain the execution behavior of multilingual models in terms of instruction format, response pattern, and task constraints, and they fail to solve the dual manifestation problem of catastrophic forgetting - not only is it manifested as language ability degradation, but also as task execution behavior drift. The continuous introduction of low-resource languages ​​and changes in their training distribution may gradually weaken the model's original language capabilities, language representations, and stable generation behavior. Essentially, it is a loss of overall capabilities caused by the adaptation of new languages ​​and changes in training distribution.

[0009] In summary, low-resource multilingual large language model adaptation techniques not only need to improve the generation quality of target language task pairs but also need to ensure the output stability of preserved language task pairs. Specifically, such methods should be able to improve task effectiveness, language consistency, and format compliance when there is insufficient supervised data for the target language task pairs, and reduce output format shifts, instruction constraint failures, and response pattern drifts in preserved language task pairs during continuous adaptation. Existing methods lack a unified technical path that simultaneously addresses task capability completion and behavioral stability maintenance, making it difficult to develop verifiable, reusable, and scalable low-resource multilingual model adaptation solutions. Summary of the Invention

[0010] To address the aforementioned technical problems in the prior art, namely, the problem of adapting low-resource multilingual models that simultaneously possess task capability completion and behavior stability maintenance, embodiments of this application provide a method, system, and electronic device for cross-language same-task capability transfer and behavior maintenance of low-resource multilingual large language models.

[0011] In a first aspect of this application, a method for cross-lingual, same-task ability transfer and behavior preservation of a low-resource, multilingual, large language model is provided, comprising:

[0012] Freeze the original parameters, routing parameters, unselected expert parameters, and shared dense layer parameters of the pre-trained dynamic hybrid expert backbone model, and construct the language codes for each language, the task codes for each task family, and the language group codes for each language group.

[0013] For each expert branch in the hybrid expert layer of the dynamic hybrid expert backbone model, based on the task gradient generated by different language task pairs being routed to the same expert branch and the corresponding routing ratio, the conflict score of the expert branch is calculated. The conflict scores of each expert branch are then sorted from high to low, and the top experts are selected. The expert branches corresponding to each conflict score constitute a high-conflict expert set H;

[0014] For each expert branch and its projection matrix q in the high-conflict expert set H, calculate the effective projection matrix, which is obtained by linearly superimposing the frozen original projection weights, shared residual branches, language group residual branches, and task-private residual branches.

[0015] Using the language encoding, the task encoding, and the language group encoding as input, the predicted task-private compact coefficient vector of the target language task pair is obtained through the task-private capability transmitter. And based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task-private residual branch, wherein the regression objective of the stopping gradient during the training phase of the task-private capability transmitter is the task-private compactness coefficient corresponding to the observed language task pair. The observed language task pairs have corresponding task-private compactness coefficients. The observed language task pairs are obtained through supervised training and learning. They belong to the observed set O, and the target language task pairs belong to the target transfer set P. The observed set O contains language task pairs with supervised sample data, and the target transfer set P contains language task pairs with unsupervised sample data or with a supervised sample data amount less than a preset threshold.

[0016] Using the stable model checkpoint of the dynamic hybrid expert backbone model prior to the adaptation phase as the teacher model, and the dynamic hybrid expert backbone model after the adaptation phase as the student model, a given retention set is used. Simultaneously, temperature-scaled KL divergence is used as the output-level distillation loss, serving as the input to both the teacher and student models. Output-level distillation is performed, and N layers of hybrid expert layers corresponding to the teacher model and the student model are selected as distillation alignment layers. For each distillation alignment layer, a loss based on the hidden state-level representation preservation is applied. The teacher model and the student model are aligned by distillation of the hidden state. The final sequence hidden state is the hidden state obtained after the output of the feedforward expert module in the distillation alignment layer is connected by residual connection. The retention set contains language task pairs used to constrain and evaluate the original language ability, output format, instruction compliance and response pattern stability of the model.

[0017] Calculate the subspace orthogonality regularization loss based on the subspace overlap of any two low-rank basis matrices with the same row dimension. And the task supervision loss is based on the standard causal language model supervision loss. ;

[0018] The dynamic hybrid expert backbone model is trained with the goal of minimizing the global loss, where the global loss is the output-level distillation loss. The hidden state level represents the retention loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum.

[0019] Optionally, the language encoding is obtained by a language encoder through parameter deviation statistical features generated by short-step monolingual adaptation, the task encoding is a learnable task embedding vector, and the language group encoding is obtained based on the centroid vectors of language encodings under the same language family, the same writing system, or linguistic similarity.

[0020] Optionally, calculating the conflict score of the expert branch based on the task gradients generated by different language task pairs being routed to the same expert branch and the corresponding routing ratios includes:

[0021] For each valid conflict analysis pair, compute the two language task pairs in the valid conflict analysis pair. and Conflict values ​​between task gradient directions Among them, when the language task is language task pair They are different, correspond to different task families, and and The routing ratio on the same expert is greater than or equal to a preset threshold. hour, and To form an effective conflict analysis pair, , and Representing language, and Indicates a task family. Indicates expert branch, Indicates the hybrid expert layer number. Indicates the first Expert IDs in a hybrid expert layer Representational language task pairs In the expert branch The task gradient on top, Representational language task pairs In the expert branch The task gradient is expressed as max(), which represents taking the maximum value. Vectorization is represented. Indicates the inner product. Represents the L2 norm, It is a very small positive number, used to prevent the denominator from being zero;

[0022] For each expert branch Computational expert branch The average of the conflict values ​​of all valid conflict analysis pairs is used to obtain the expert branch. Conflict score If expert branches If no corresponding valid conflict analysis pair exists, then the conflict score of the expert branch will be... Set to 0.

[0023] The shared residual branch is used to capture cross-language general adaptation signals, the language group residual branch is used to capture adaptation signals of language family, writing system or language type, and the task-private residual branch is used to carry the transferable task execution behavior of the task family. The task-private residual branch is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair.

[0024] Optionally, the effective projection matrix ;

[0025] in, Let represent the expert branch, and q represent the projection matrix of the expert branch. Represents the frozen original projection weights;

[0026] Indicates a shared residual branch. , The low-rank projection matrix represents the shared residual branch. Represents the low-rank reconstruction matrix of the shared residual branches;

[0027] Represents the residual branches of language groups. , The low-rank projection matrix represents the residual branch of the language group. This represents the low-rank reconstruction matrix of the residual branches of the language group, where g is the language group index. This indicates the language group to which language u belongs; all languages ​​within the same language group share the same set of low-rank parameters.

[0028] This represents a task-private residual branch, which is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair. , and These are the first and second low-rank basis matrices shared across language groups t, respectively. Represents a diagonal function. For observed language task pairs The task-private compact coefficient vector is used to control the combination weights and activation strengths of the task behavior subspace under the combination of language u and task family t.

[0029] Optionally, the cross-language same-task capability transfer and behavior preservation method of the low-resource multilingual large language model uses the concatenation result of the language encoding, the task encoding, and the language group encoding as input, and obtains the prediction task-private compact coefficient vector of the target language task pair through a task-private capability transmitter. ,in, Representation Language Language encoding, Represents task family Task coding, Representation Language The language group code of the language group to which it belongs. Indicates splicing, This refers to a task-private capability transmitter, and the training loss during the training phase of the task-private capability transmitter. , This indicates that the gradient operation is stopped. This represents the task-private compactness coefficient of the observed language task.

[0030] In obtaining the private compact coefficient vector of the prediction task for the target language task pair Then, based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task's private residual branch .

[0031] Optional, output stage distillation loss ,in, This indicates the output stage distillation temperature coefficient. This represents the total number of samples in the retained set. To preserve individual sample pairs in the set, Given the input sequence, For the corresponding target output sequence, Indicates the length of the token in the target output sequence. The token position index in the target output sequence, with a value range of 1. , This represents all prefix tokens preceding the j-th token in the target output sequence. Teacher model At temperature Given input and prefix The next token prediction distribution at that time Student model At temperature Given input and prefix The next token prediction distribution at that time Indicates KL divergence;

[0032] Hidden state level representation maintains loss ,in, For a pre-selected set of distillation-aligned layers, The number of distillation alignment layers, For the input sequence The length of the token. Teacher model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Student model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Denotes the square of the Frobenius norm;

[0033] Total Distillation Loss For output stage distillation losses And hidden state level representation maintain loss The weighted sum, ,in, Distillation weights for hidden states;

[0034] Subspace overlap of any two low-rank basis matrices X and Y with the same row dimension ,in, and The subspace orthogonal regularization loss is the matrix obtained by normalizing the column vectors of the low-rank basis matrices X and Y, respectively. The average value of the overlap of all subspaces over the current training language task pair, the selected set of high-conflict experts, and the set of projection matrices;

[0035] Task monitoring loss ,in, This represents the batch of current training language task pairs sampled from the new task dataset. This represents the total number of samples in the batch for the current language task training. Student model The set of trainable parameters Indicates the student model in parameters Given input and prefix Predict the next token as The probability of;

[0036] The global loss is the distillation loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum, global loss ,in, For the preset orthogonal regularization weights, This is the preset total distillation loss weight.

[0037] Optionally, the method for cross-language same-task ability transfer and behavior preservation in the low-resource multilingual large language model further includes:

[0038] For target language task pairs, perplexity, ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence), chrF (CHaRacter-level F-score), target language consistency, and task effectiveness are calculated to assess the degree of task capability completion. If the task effectiveness is lower than a preset effectiveness threshold, the parameters of the task private capability transmitter are readjusted, and the predicted task private compact coefficient vector of the target language task pair is recalculated.

[0039] For language task pairs in the retention set, the format compliance rate, instruction constraint satisfaction rate, response pattern consistency rate, and hidden state drift distance are calculated to evaluate the catastrophic forgetting suppression effect. If the format compliance rate is lower than a preset compliance threshold or the instruction constraint satisfaction rate is lower than a preset constraint threshold, the total distillation loss weight is increased or the retention set sampling ratio is increased.

[0040] In a second aspect of this application, a cross-lingual, same-task ability transfer and behavior preservation system for a low-resource, multilingual, large language model is provided, comprising:

[0041] The data construction and encoding module is used to freeze the original parameters, routing parameters, unselected expert parameters, and shared dense layer parameters of the pre-trained dynamic hybrid expert backbone model, and to construct language codes for each language, task codes for each task family, and language group codes for each language group. It also constructs an observation set O and a target transfer set P. Observed language task pairs belong to the observation set O, and target language task pairs belong to the target transfer set P. The observation set O contains language task pairs with supervised sample data, and the target transfer set P contains language task pairs with unsupervised sample data or with a supervised sample data amount less than a preset threshold.

[0042] The high-conflict expert screening module is used to calculate the conflict score of each expert branch in each hybrid expert layer of the dynamic hybrid expert backbone model, based on the task gradient generated by different language task pairs being routed to the same expert branch and the corresponding routing ratio. The module then sorts the conflict scores of each expert branch from high to low and filters out the top-performing experts. The expert branches corresponding to each conflict score constitute a high-conflict expert set H;

[0043] A combined low-rank parameter decoupling module is used to construct shared residual branches, language group residual branches, and task-private residual branches, and to calculate an effective projection matrix for each expert branch and its projection matrix q in the high-conflict expert set H. The effective projection matrix is ​​obtained by linearly superimposing the frozen original projection weights, shared residual branches, language group residual branches, and task-private residual branches.

[0044] The task-private capability transmission module is used to obtain the prediction task-private compact coefficient vector of the target language task pair by using the language encoding, the task encoding, and the language group encoding as input through the task-private capability transmitter. And based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task-private residual branch, wherein the regression objective of the stopping gradient during the training phase of the task-private capability transmitter is the task-private compactness coefficient corresponding to the observed language task pair. The observed language task pairs have corresponding task-private compactness coefficients. Acquired through supervised training;

[0045] The behavior retention and forgetting inhibition module is used for:

[0046] Using the stable model checkpoint of the dynamic hybrid expert backbone model prior to the adaptation phase as the teacher model, and the dynamic hybrid expert backbone model after the adaptation phase as the student model, a given retention set is used. Simultaneously, temperature-scaled KL divergence is used as the output-level distillation loss, serving as the input to both the teacher and student models. Output-level distillation is performed, and N layers of hybrid expert layers corresponding to the teacher model and the student model are selected as distillation alignment layers. For each distillation alignment layer, a loss based on the hidden state-level representation preservation is applied. The teacher model and the student model are aligned by distillation of the hidden state. The final sequence hidden state is the hidden state obtained after the output of the feedforward expert module in the distillation alignment layer is connected by residual connection. The retention set contains language task pairs used to constrain and evaluate the original language ability, output format, instruction compliance and response pattern stability of the model.

[0047] Calculate the subspace orthogonality regularization loss based on the subspace overlap of any two low-rank basis matrices with the same row dimension. And the task supervision loss is based on the standard causal language model supervision loss. ;

[0048] The dynamic hybrid expert backbone model is trained with the goal of minimizing the global loss, where the global loss is the output-level distillation loss. The hidden state level represents the retention loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum.

[0049] Optionally, the language encoding is obtained by a language encoder through parameter deviation statistical features generated by short-step monolingual adaptation, the task encoding is a learnable task embedding vector, and the language group encoding is obtained based on the centroid vectors of language encodings under the same language family, the same writing system, or linguistic similarity.

[0050] Optionally, the high-conflict expert screening module, when calculating the conflict score of the expert branch based on the task gradients generated by different language task pairs being routed to the same expert branch and the corresponding routing ratios, is specifically used for:

[0051] For each valid conflict analysis pair, compute the two language task pairs in the valid conflict analysis pair. and Conflict values ​​between task gradient directions Among them, when the language task is language task pair They are different, correspond to different task families, and and The routing ratio on the same expert is greater than or equal to a preset threshold. hour, and To form an effective conflict analysis pair, , and Representing language, and Indicates a task family. Indicates expert branch, Indicates the hybrid expert layer number. Indicates the first Expert IDs in a hybrid expert layer Representational language task pairs In the expert branch The task gradient on top, Representational language task pairs In the expert branch The task gradient is expressed as max(), which represents taking the maximum value. Vectorization is represented. Indicates the inner product. Represents the L2 norm, It is a very small positive number, used to prevent the denominator from being zero;

[0052] For each expert branch Computational expert branch The average of the conflict values ​​of all valid conflict analysis pairs is used to obtain the expert branch. Conflict score If expert branches If no corresponding valid conflict analysis pair exists, then the conflict score of the expert branch will be... Set to 0.

[0053] Optionally, the combined low-rank parameter decoupling module is specifically used for:

[0054] Calculate the effective projection matrix ;

[0055] in, Let represent the expert branch, and q represent the projection matrix of the expert branch. Represents the frozen original projection weights;

[0056] Indicates a shared residual branch. , The low-rank projection matrix represents the shared residual branch. Represents the low-rank reconstruction matrix of the shared residual branches;

[0057] Represents the residual branches of language groups. , The low-rank projection matrix represents the residual branch of the language group. This represents the low-rank reconstruction matrix of the residual branches of the language group, where g is the language group index. This indicates the language group to which language u belongs; all languages ​​within the same language group share the same set of low-rank parameters.

[0058] This represents a task-private residual branch, which is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair. , and These are the first and second low-rank basis matrices shared across language groups t, respectively. Represents a diagonal function. For observed language task pairs The task-private compact coefficient vector is used to control the combination weights and activation strengths of the task behavior subspace under the combination of language u and task family t.

[0059] The shared residual branch is used to capture cross-language general adaptation signals, the language group residual branch is used to capture adaptation signals of language family, writing system or language type, and the task-private residual branch is used to carry the transferable task execution behavior of the task family. The task-private residual branch is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair.

[0060] Optionally, the task private capability transmission module is specifically used for:

[0061] Using the concatenation result of the language encoding, the task encoding, and the language group encoding as input, the predicted task-private compact coefficient vector of the target language task pair is obtained through the task-private capability transmitter. ,in, Representation Language Language encoding, Represents task family Task coding, Representation Language The language group code of the language group to which it belongs. Indicates splicing, This refers to a task-private capability transmitter, and the training loss during the training phase of the task-private capability transmitter. , This indicates that the gradient operation is stopped. This represents the task-private compactness coefficient of the observed language task.

[0062] In obtaining the private compact coefficient vector of the prediction task for the target language task pair Then, based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task's private residual branch .

[0063] Optionally, the behavior retention and forgetting inhibition module is specifically used for:

[0064] Calculate output stage distillation losses ,in, This indicates the output stage distillation temperature coefficient. This represents the total number of samples in the retained set. To preserve individual sample pairs in the set, Given the input sequence, For the corresponding target output sequence, Indicates the length of the token in the target output sequence. The token position index in the target output sequence, with a value range of 1. , This represents all prefix tokens preceding the j-th token in the target output sequence. Teacher model At temperature Given input and prefix The next token prediction distribution at that time Student model At temperature Given input and prefix The next token prediction distribution at that time Indicates KL divergence;

[0065] Calculate the hidden state level representation retention loss ,in, For a pre-selected set of distillation-aligned layers, The number of distillation alignment layers, , For the input sequence The length of the token. Teacher model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Student model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Denotes the square of the Frobenius norm;

[0066] Calculate total distillation loss For output stage distillation losses And hidden state level representation maintain loss The weighted sum, ,in, Distillation weights for hidden states;

[0067] Calculate the subspace overlap of any two low-rank basis matrices X and Y with the same row dimension. ,in, and The subspace orthogonal regularization loss is the matrix obtained by normalizing the column vectors of the low-rank basis matrices X and Y, respectively. The average value of the overlap of all subspaces over the current training language task pair, the selected set of high-conflict experts, and the set of projection matrices;

[0068] Calculate the monitoring loss of the task ,in, This represents the batch of current training language task pairs sampled from the new task dataset. This represents the total number of samples in the batch for the current language task training. Student model The set of trainable parameters Indicates the student model in parameters Given input and prefix Predict the next token as The probability of;

[0069] Calculate the global loss, which is the distillation loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum, global loss ,in, For the preset orthogonal regularization weights, This is the preset total distillation loss weight;

[0070] The dynamic hybrid expert backbone model is trained with the goal of minimizing the global loss.

[0071] Optionally, the low-resource multilingual large language model cross-language same-task ability transfer and behavior preservation system further includes an evaluation and feedback module, which is used for:

[0072] For target language task pairs, calculate perplexity, ROUGE-L, chrF, target language consistency, and task effectiveness to assess the degree of task capability completion. If the task effectiveness is lower than a preset effectiveness threshold, readjust the parameters of the task private capability transmitter and recalculate the predicted task private compactness coefficient vector of the target language task pair.

[0073] For language task pairs in the retention set, the format compliance rate, instruction constraint satisfaction rate, response pattern consistency rate, and hidden state drift distance are calculated to evaluate the catastrophic forgetting suppression effect. If the format compliance rate is lower than a preset compliance threshold or the instruction constraint satisfaction rate is lower than a preset constraint threshold, the total distillation loss weight is increased or the retention set sampling ratio is increased.

[0074] In a third aspect of this application, an electronic device is provided, the electronic device comprising:

[0075] At least one processor; and,

[0076] A memory communicatively connected to at least one of the processors; wherein,

[0077] The memory stores instructions that can be executed by the processor to implement the aforementioned method for cross-language, same-task capability transfer and behavior preservation of a low-resource, multilingual, large language model.

[0078] The cross-language, same-task capability transfer and behavior preservation scheme for low-resource, multilingual, large language models provided in this application has the following beneficial effects:

[0079] (1) Effectively reduces the construction cost of supervision data for target language tasks. By predicting the compactness coefficient of target language task pairs through the task private capability transmitter, task execution behavior can be efficiently instantiated under the condition of no supervision data for target language task pairs or only a very small amount of such supervision data. This significantly reduces the dependence on translation data and manually annotated data, and reduces the construction cost and implementation difficulty of supervision data in low-resource languages, minority languages ​​and language scenarios with large differences in writing systems.

[0080] (2) Significantly improve the ability transfer effect across languages ​​and tasks. By innovatively modeling task execution behavior as composable task-private capabilities, and leveraging the synergistic effect of the shared base of task families and the compactness coefficient of language task pairs, the cross-language reuse of task execution behavior is realized. This ensures that the task knowledge, feature representation and execution methods learned in high-resource language task pairs can be efficiently and accurately transferred to low-resource target language task pairs, greatly improving the execution quality and efficiency of low-resource language tasks.

[0081] (3) Effectively alleviates gradient interference problem in multilingual and multi-task training. By accurately locating high-conflict expert branches, combined low-rank parameter decoupling operation is performed only in the region where gradient conflict is most significant, avoiding irrelevant disturbances caused by full model update or arbitrary expert update, reducing mutual cancellation and destruction of parameter updates during multilingual and multi-task training, and significantly improving the stability and reliability of low-resource language task transfer.

[0082] (4) Effectively suppress catastrophic forgetting phenomenon. By adopting a dual suppression strategy that combines output-level distillation and hidden-state distillation, output-level distillation stabilizes the model's response distribution, output format and instruction compliance pattern, while hidden-state distillation stabilizes the internal feature representation of the multilingual model. The two work together to maintain the model's original language capabilities and task execution behavior, thus solving the problem of overall capability loss in the process of low-resource language adaptation from the root.

[0083] (5) The parameters are highly efficient and easy to deploy. By freezing the backbone model, routing parameters, unselected experts and shared dense layers, only a small number of low-rank residual branches, compact coefficients and transmitter parameters are trained, which greatly reduces the training, storage and deployment costs of the model. It can be quickly extended to more low-resource languages ​​and tasks under limited computing power conditions, and adapt to large-scale and diversified multilingual application scenarios.

[0084] (6) It has good verifiability and scalability. It can be quantitatively verified through task performance indicators, behavior retention indicators, etc., to ensure the effectiveness and reliability of the technical solution. It can also be extended to more languages, task families, model scales and continuous learning scenarios, and has broad application prospects and promotion value. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 A flowchart illustrating the method for cross-language, same-task capability transfer and behavior preservation in a low-resource, multilingual, large language model provided in this application embodiment;

[0087] Figure 2 The diagram below shows the structure of a low-resource, multilingual, large language model cross-language same-task capability transfer and behavior preservation system provided in this application embodiment. Detailed Implementation

[0088] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0089] The following will describe in detail, with reference to the accompanying drawings, a method for cross-language, same-task capability transfer and behavior preservation of a low-resource, multilingual, large language model according to an embodiment of this application. For example... Figure 1 As shown, the method for cross-language same-task capability transfer and behavior preservation of a low-resource multilingual large language model according to the first embodiment of this application includes:

[0090] Step S101: Freeze the original parameters, routing parameters, unselected expert parameters, and shared dense layer parameters of the pre-trained dynamic hybrid expert backbone model, and construct the language codes for each language, the task codes for each task family, and the language group codes for each language group.

[0091] Step S102: For each expert branch in the hybrid expert layer of the dynamic hybrid expert backbone model, based on the task gradient generated by different language task pairs being routed to the same expert branch and the corresponding routing ratio, calculate the conflict score of the expert branch, sort the conflict scores of each expert branch from high to low, and select the top... The expert branches corresponding to each conflict score constitute a high-conflict expert set H;

[0092] Step S103: For each expert branch and its projection matrix q in the high-conflict expert set H, calculate the effective projection matrix. The effective projection matrix is ​​obtained by linearly superimposing the frozen original projection weights, shared residual branches, language group residual branches, and task-private residual branches.

[0093] Step S104: Using the language encoding, the task encoding, and the language group encoding as input, obtain the prediction task-private compact coefficient vector of the target language task pair through the task-private capability transmitter. And based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task-private residual branch, wherein the regression objective of the stopping gradient during the training phase of the task-private capability transmitter is the task-private compactness coefficient corresponding to the observed language task pair. The observed language task pairs have corresponding task-private compactness coefficients. Acquired through supervised training;

[0094] Step S105: Using the stable model checkpoint of the dynamic hybrid expert backbone model before the adaptation phase as the teacher model, and the dynamic hybrid expert backbone model after the adaptation phase as the student model, using the given retention set. Simultaneously, temperature-scaled KL divergence is used as the output-level distillation loss, serving as the input to both the teacher and student models. Output-level distillation is performed, and N layers of hybrid expert layers corresponding to the teacher model and the student model are selected as distillation alignment layers. For each distillation alignment layer, a loss based on the hidden state-level representation preservation is applied. The teacher model and the student model are aligned by distillation of the hidden state. The final sequence hidden state is the hidden state obtained after the output of the feedforward expert module in the distillation alignment layer is connected by residual connection.

[0095] Step S106: Calculate the subspace orthogonal regularization loss based on the subspace overlap of any two low-rank basis matrices with the same row dimension. And the task supervision loss is based on the standard causal language model supervision loss. ;

[0096] Step S107: Train the dynamic hybrid expert backbone model with the optimization objective of minimizing the global loss, where the global loss is the output-level distillation loss. The hidden state level represents the retention loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum.

[0097] Specifically, in step S101, the dynamic hybrid expert backbone model can be based on the Transformer architecture and includes several sparse routing expert layers. The dynamic hybrid expert backbone model can be a Transformer large language model, a dense large language model, or different types of dynamic hybrid expert models. The number of hybrid expert layers, the number of experts, and the routing strategy can be adjusted according to the model size. For example, the routing strategy can be top-1 routing, top-2 routing, or other sparse routing strategies. In step S101, the original parameters, routing parameters, unselected expert parameters, and shared dense layer parameters of the pre-trained dynamic hybrid expert backbone model are frozen to ensure that these parts remain unchanged during subsequent adaptation, thereby avoiding disturbances to the original language capabilities and task execution behavior caused by parameter updates. Simultaneously, language codes for each language are constructed. These codes are calculated by a language encoder based on the parameter deviation statistical features generated during short-step monolingual adaptation, or they can be obtained from the mean of language corpus representation, vocabulary coverage features, writing system features, typological features, or multi-source language embedding fusion. Learnable task embedding vectors are constructed for each task family as task codes. Based on the centroid vectors of language codes under the same language family, writing system, or linguistic similarity, language group codes for each language group are generated. Language groups can be determined by language family, writing system, geographical region, morphological type, character set similarity, or model internal representation clustering results. In addition, in step S101, observed language task pairs and target language task pairs can be divided to construct an observed set O and a target transfer set P. Observed language task pairs belong to the observed set O, and target language task pairs belong to the target transfer set P. The observed set O contains language task pairs with supervised sample data, while the target transfer set P contains language task pairs with unsupervised sample data or very small supervised sample data (e.g., data volume less than a preset threshold).

[0098] Specifically, in step S102, for each expert branch of each hybrid expert layer in the dynamic hybrid expert backbone model, based on the task gradients generated when different language task pairs are routed to the same expert branch and their corresponding routing ratios, the conflict score of that expert branch is calculated to identify the expert regions most prone to parameter interference during multilingual multi-task training. Specifically, the language task pairs... In the expert branch The task gradient on is denoted as , to language task The routing ratio on this expert branch is denoted as ,in, Representing language, Indicates a task family. Indicates the hybrid expert layer number. Indicates the first Expert IDs in a hybrid expert layer This represents the batch of current training language task pairs sampled from the new task dataset. This indicates that samples in this batch were routed to experts. The token ratio. For each expert branch, if it simultaneously meets two conditions, it is defined as a high-conflict expert branch: Condition 1, the proportion of multiple language tasks (e.g., more than two) being routed to this expert branch reaches a preset threshold; Condition 2, there is a significant conflict in the gradient directions of different task families on this expert branch. Specifically, when language task pairs... language task pair They are different, correspond to different task families, and and The routing ratio on the same expert is greater than or equal to a preset threshold. At that time, that is , hour, and To form an effective conflict analysis pair, where, Representing language, Indicates a task family. Representational language task pairs The routing ratio on the expert branch; for each valid conflict analysis pair, calculate the two language task pairs in the valid conflict analysis pair. and Conflict values ​​between task gradient directions ,in, The cosine similarity between the gradient vectors of two tasks is represented by... When the gradient directions of two tasks are opposite, the cosine similarity is negative, indicating that they have interfering update directions on the same expert branch. Therefore, we take the opposite number and retain the non-negative conflict strength through the max() function. When the gradient directions of two tasks are consistent or nearly consistent, the conflict value is set to 0. Representational language task pairs In the expert branch The task gradient is expressed as max(), which represents taking the maximum value. Vectorization is represented. Indicates the inner product. Represents the L2 norm, It is a very small positive number, used to prevent the denominator from being zero. For each expert branch Computational expert branch The average of the conflict values ​​of all valid conflict analysis pairs is used to obtain the expert branch. Conflict score If expert branches If no corresponding valid conflict analysis pair exists, then the conflict score of the expert branch will be... Set it to 0. Then sort the conflict scores of each expert branch from high to low, and filter out the top... The expert branches corresponding to the conflict scores constitute a high-conflict expert set H. Through the above screening mechanism, it is not necessary to update all expert branches or all model parameters indiscriminately. Instead, the expert branches with the most significant task gradient interference in multilingual multi-task training (high-conflict expert branches) are prioritized. Subsequently, shared residual branches, language group residual branches, and task-specific residual branches are introduced only in the high-conflict regions corresponding to these high-conflict expert branches. This reduces irrelevant parameter perturbations, improves the stability of the target language task for transfer, and reduces the disruption to the task execution behavior of the preserved language task pairs during new language adaptation. In one possible implementation, the conflict scores of expert branches can also be expressed as gradient projection conflict, gradient angle, gradient inner product, or other statistics reflecting task interference. It can be a preset fixed value, or it can be adaptively determined according to the number of model layers, the total number of experts, or the conflict distribution.

[0099] Specifically, in step S103, a low-rank residual update term is introduced outside the original frozen projection matrix of the high-conflict expert branch. This low-rank residual structure consists of a shared residual branch, a language group residual branch, and a task-specific residual branch. The shared residual branch is used to capture cross-language general adaptation signals; the language group residual branch is used to capture adaptation signals of language families, writing systems, or language types; and the task-specific residual branch carries the transferable task execution behavior of the task family. The task-specific residual branch is obtained based on the shared low-rank basis matrix of the task family and the task-specific compact coefficient vector of the corresponding language task pair. The specific forms of the shared residual branch, language group residual branch, and task-specific residual branch can adopt LoRA (Low-Rank Adaptation of Large Language Models) low-rank decomposition, adapter modules, gated residual modules, tensor decomposition modules, or other parametrically efficient structures. These structures are based on existing technologies and will not be elaborated further. Instead of directly modifying the original frozen projection weights of the expert branches, multiple low-rank residual branches are superimposed on them to form an effective projection matrix, which is a combined low-rank parameter decoupling. Specifically, for each expert branch and its projection matrix q in the high-conflict expert set H, an effective projection matrix is ​​calculated. This effective projection matrix is ​​obtained by linearly superimposing the frozen original projection weights, shared residual branches, language group residual branches, and task-private residual branches. Where q represents the projection matrix of the expert branch, Represents the frozen original projection weights;

[0100] Indicates a shared residual branch. , The low-rank projection matrix represents the shared residual branch. Represents the low-rank reconstruction matrix of the shared residual branches;

[0101] Represents the residual branches of language groups. , The low-rank projection matrix represents the residual branch of the language group. This represents the low-rank reconstruction matrix of the residual branches of the language group, where g is the language group index. This indicates the language group to which language u belongs; all languages ​​within the same language group share the same set of low-rank parameters.

[0102] This represents a task-private residual branch, which is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair. , and These are the first and second low-rank basis matrices shared across language groups t, respectively. Represents a diagonal function. For observed language task pairs The task-private compact coefficient vector is used to control the combination weights and activation strengths of the task behavior subspace under the combination of language u and task family t.

[0103] Specifically, in step S104, for the observed set Observed language tasks in First, the corresponding task-specific compactness coefficient is directly learned through supervised training. , This indicates that observed language task pairs share a basis matrix within the task family. and The combined weights are then fixed using the trained task family basis matrix. and And assign the observed language task to the corresponding task-private compactness coefficient. As the regression target for stopping gradients, the training task's private capability transmitter Mission Private Capability Transmitter A two-layer MLP (Multi-Layer Perceptron), residual MLP, Transformer encoder, gated network, or small hybrid expert network can be used. The training objective can be mean squared error, and contrastive learning loss, distribution matching loss, or confidence calibration term can be incorporated into the training objective. Then, the language encoding, task encoding, and language group encoding are used as inputs. Specifically, the language encoding, task encoding, and language group encoding are sequentially concatenated to form the task-specific capability transmitter. The input is processed by the task-private capability transmitter to obtain the prediction task-private compact coefficient vector of the target language task pair. , ,in, Representation Language Language encoding, Represents task family Task coding, Representation Language The language group code of the language group to which it belongs. This indicates splicing, and the training loss during the training phase of the task-private capability transmitter. , This indicates that the gradient operation is stopped. This represents the observed language task's task-private compactness coefficient. Then... Substitute the private residual branch formula for the task: This enables the instantiation of task capabilities under conditions where there is no target language task pair supervision data or only a very small amount of target language task pair supervision data.

[0104] Specifically, in step S105, to effectively mitigate the catastrophic forgetting caused by low-resource adaptation, a joint distillation mechanism of the teacher model and the student model is introduced. The stable model checkpoint of the dynamic hybrid expert backbone model before the adaptation phase is used as the teacher model, and the dynamic hybrid expert backbone model after the adaptation phase is used as the student model, using a given retention set. Simultaneously serving as input to both the teacher and student models, the retention set contains language task pairs used to constrain and evaluate the stability of the models' original language abilities, output formats, instruction compliance, and response patterns. Temperature-scaled KL divergence is used as the output-level distillation loss. Output stage distillation, distillation loss ,in, This indicates the output stage distillation temperature coefficient. This represents the total number of samples in the retained set. To preserve individual sample pairs in the set, Given the input sequence, For the corresponding target output sequence, Indicates the length of the token in the target output sequence. The token position index in the target output sequence, with a value range of 1. , This represents all prefix tokens preceding the j-th token in the target output sequence. Teacher model At temperature Given input and prefix The next token prediction distribution at that time Student model At temperature Given input and prefix The next token prediction distribution at that time The KL divergence is used to preserve the response distribution, format constraints, and instruction compliance patterns of the dynamic hybrid expert backbone model. N hybrid expert layers corresponding to the teacher and student models are selected as distillation alignment layers. For each distillation alignment layer, the loss is preserved based on the hidden state level representation. The final sequence hidden states corresponding to the teacher model and the student model are aligned by hidden state distillation. These final sequence hidden states are the hidden states obtained after the output of the feedforward expert module in the distillation alignment layer is passed through a residual connection. By aligning the final sequence hidden states, the internal multilingual representation can be stabilized without relying on specific routing paths, reducing representation drift caused by new language adaptation. Specifically, hidden state-level representation preservation loss... ,in, For a pre-selected set of distillation-aligned layers, The number of distillation alignment layers, , For the input sequence The length of the token. Teacher model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Student model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, This represents the square of the Frobenius norm. Total distillation loss. For output stage distillation losses And hidden state level representation maintain loss The weighted sum, ,in, The weights are used for distilling the hidden states. In one possible implementation, the teacher model can also be the original dynamic hybrid expert backbone model or a stable average model; the distillation alignment layer can be one or more hybrid expert layers, or it can be adaptively selected according to the conflict distribution.

[0105] Specifically, in step S106, to prevent the shared residual branch, language group residual branch, and task-private residual branch from learning overlapping or collapsed parameter subspaces, subspace orthogonal regularization is introduced. The subspace orthogonal regularization loss is calculated based on the subspace overlap of any two low-rank basis matrices with the same row dimension. Specifically, the subspace overlap of any two low-rank basis matrices X and Y with the same row dimension. ,in, and The subspace orthogonal regularization loss is the matrix obtained by normalizing the column vectors of the low-rank basis matrices X and Y, respectively. This is the average overlap of all subspaces over the current training language task pair, the selected set of high-conflict experts, and the projection matrix set. Specifically, in step S106, the task supervision loss is set using the standard causal language model supervision loss. Task supervision loss ,in, This represents the batch of current training language task pairs sampled from the new task dataset. This represents the total number of samples in the batch for the current language task training. Student model The set of trainable parameters Indicates the student model in parameters Given input and prefix Predict the next token as The probability of.

[0106] Specifically, in step S106, the global loss is calculated, the global loss The total distillation loss The subspace orthogonal regularization loss and the task supervision loss The weighted sum, i.e., the global loss is the output stage distillation loss. The hidden state level represents the retention loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum, specifically, Then, with minimizing the global loss as the optimization objective, the dynamic hybrid expert backbone model is trained. This guides the model to simultaneously achieve comprehensive optimization in response behavior stability, internal representation consistency, subspace decoupling, and task performance maintenance during the training process. This enables the model to retain old capabilities while learning new tasks and achieve effective knowledge transfer.

[0107] In summary, the cross-lingual, same-task ability transfer and behavior preservation method for low-resource, multilingual, large language models provided in this application can be divided into five stages:

[0108] In the first stage, a dynamic hybrid expert backbone model is trained or acquired, and its frozen backbone model parameters, routing parameters, unselected expert parameters, and shared dense layer parameters are frozen.

[0109] In the second stage, high-conflict expert branch analysis is performed to identify the set of high-conflict experts H that needs to be decoupled from parameters. The model parameters are not updated in this stage.

[0110] The third stage involves the observed set. Language tasks with supervised data Based on freezing the backbone model parameters, routing parameters, unselected expert parameters, and shared dense layer parameters, only the shared residual branches and language group residual branches in the high-conflict expert branches are updated, as well as the low-rank basis matrix shared with the task family and the task-private compactness coefficients corresponding to the observed language task pairs. The relevant task-specific residual branches, specifically, the trainable parameters include , , , and ,in, As the stopping gradient regression objective of the training phase of the task-private capability transmitter, the third phase includes the observed language task pair subspace learning phase. The goal is to learn the shared adaptation subspace, the language group adaptation subspace, and the task-private behavior subspace, so that the task execution behavior in the observed language task pairs can be represented as composable and transferable low-rank capability units.

[0111] In the fourth stage, the parameters learned in the third stage are frozen, and the task's private capability transmitter is trained. Update the language encoding, task encoding, language group encoding, and task-private capability transmitter;

[0112] The fifth stage, the target language task transfer and behavior preservation stage, instantiates the task-private residual branch using the task-private capability transmitter's predicted task-private compactness coefficients (obtaining the predicted task-private compactness coefficients). Output-level and hidden-state-level distillation constraints are applied based on the retained set to preserve the original language capabilities, internal representations, and stable task execution behavior. This is achieved through output-level distillation loss. The hidden state level represents the retention loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum of the global loss is used to optimize the training of a dynamic hybrid expert backbone model.

[0113] After training, the dynamic hybrid expert backbone model can be used for output inference.

[0114] In one possible implementation, to verify the effectiveness of cross-language task transfer and behavior preservation, the cross-language same-task ability transfer and behavior preservation method for low-resource multilingual large language models provided in this application embodiment further includes an evaluation feedback mechanism, which specifically includes:

[0115] For target language task pairs, perplexity, ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation), chrF (ChaRacter-level F-score), target language consistency, instruction constraints, and task effectiveness are calculated to assess the degree of task capability completion. These metrics can be calculated using existing techniques and will not be elaborated further. If the task effectiveness is lower than a preset effectiveness threshold, the parameters of the task-private capability transmitter are readjusted, and the predicted task-private compactness coefficient vector for the target language task pair is recalculated. If ROUGE-L, chrF, or other metrics measuring the reference likelihood or text overlap of the target language task pair are lower than their corresponding preset thresholds, adjustments are made. Alternatively, if the target language consistency or instruction constraints are below the corresponding preset threshold, the lower rank number will be increased. Or add a reserved set constraint;

[0116] For language task pairs in the retention set (i.e., retained language task pairs), metrics such as format compliance rate, instruction constraint satisfaction rate, response pattern consistency rate, and hidden state drift distance are calculated to evaluate the catastrophic forgetting suppression effect. Among them, the format compliance rate and instruction constraint satisfaction rate can be used to measure the degree of ability decline of the retained language task pairs after adaptation. If the format compliance rate is lower than a preset compliance threshold or the instruction constraint satisfaction rate is lower than a preset constraint threshold, the total loss weight of distillation is increased or the sampling ratio of the retention set is increased.

[0117] It should be noted that the above is an illustrative explanation of the evaluation metrics and corresponding feedback operations for the target task pair and the retained task pair, and is not a limitation on the evaluation metrics and feedback operations. Users can also set other evaluation metrics and corresponding feedback operations as needed to examine the model's cross-language same-task ability transfer and behavior retention effects. In addition to the evaluation and feedback mechanisms for the target task pair and the retained task pair, metrics such as the mean and standard deviation of multiple random seeds and the routing distribution entropy can be introduced to measure the stability of the model in different training rounds. If the fluctuation is too large, the high-conflict expert branch analysis and selection will be re-executed. The above closed-loop evaluation and feedback mechanism ensures the performance and stability of the model during the continuous learning process.

[0118] The following example of a specific training configuration illustrates the implementation of the cross-lingual, same-task ability transfer and behavior preservation method for the low-resource, multilingual, large language model proposed in this application. Specifically, the pre-trained dynamic hybrid expert backbone model uses Qwen2.5-1.5B or an equivalent Transformer language model, keeping the original word segmenter unchanged and not expanding the vocabulary. In the construction of the dynamic hybrid expert backbone model, two Transformer layers can be expanded into hybrid expert layers, each containing 4 experts, using a top-1 routing approach.

[0119] The AdamW optimizer is used for model training. For example, the beta1 of the AdamW optimizer is set to 0.9, the beta2 to 0.95, the weight decay to 0.1, the maximum norm of gradient clipping to 1.0, and the learning rate uses a cosine decay strategy with a 3% warm-up ratio. Mixed precision training with bfloat16 is employed. During the training phase of the dynamic hybrid expert backbone model, the learning rate is 2e-5, the global batch size is 256 sequences, and the maximum sequence length is 2048. The learning rate during the training phase of the task-private compactness coefficients for the observed language task pairs is 1e-4; the learning rate during the training phase of the task-private capability transmitter is 5e-4; and the learning rate during the instantiation phase of the task-private residual branches is 5e-5.

[0120] In the language encoding construction, the short-step monolingual adaptation step number K can be set to 100, and the learning rate is 1e-5. Normalized Frobenius norm, mean, and standard deviation are extracted from the parameters of the last three Transformer layers as parameter deviation statistical features. The language encoder can use a two-layer MLP with a hidden dimension of 512 and an output dimension of 128; both the task encoding and language group encoding have a dimension of 128. The default low-rank number of the shared residual branch, language group residual branch, and task-specific residual branch can all be set to 8. The task-specific capability transmitter can use a two-layer MLP with a hidden dimension of 512 and the activation function is GELU (Gaussian Error Linear Unit).

[0121] In behavior-based distillation, the output stage distillation temperature coefficient can be set. =2.0, total weight of distillation loss Hidden state distillation weights Orthogonal regularized weights In the output inference phase of the dynamic hybrid expert backbone model, for the target language task pair, only the task-private compactness coefficients predicted by the task-private capability transmitter and the corresponding high-conflict expert branches are activated; for the reserved language task pair, the original output format, instruction compliance method and response mode are maintained by freezing backbone parameters, restricting updates and distillation constraints.

[0122] The cross-language, same-task capability transfer and behavior preservation method for low-resource multilingual large language models provided in this application relies on a task-private capability transmitter to predict the compactness coefficient of the target language task pair. This allows for task execution behavior instantiation under unsupervised data or with only minimal supervised data, reducing reliance on translation data and manually annotated data. This effectively lowers the difficulty and implementation cost of data construction for low-resource languages, minority languages, and languages ​​with special writing systems. By modeling task execution behavior as composable task-private capabilities and combining a shared base for task families with the compactness coefficient of language task pairs, it achieves cross-language reuse of task behavior. This enables efficient transfer of task knowledge, feature representations, and execution methods learned in high-resource languages ​​to low-resource target languages, significantly improving the task execution quality of low-resource languages. Furthermore, it accurately locates high-conflict expert branches and performs combined low-rank parameter decoupling only in regions of significant gradient conflict, avoiding invalid parameter perturbations caused by full model updates and random expert updates, thus reducing the training burden of multilingual multi-task training. The mutual interference of parameters during training effectively improves the stability of low-resource language task transfer. Combining the dual constraint mechanism of output-level distillation and hidden-state distillation, output-level distillation stabilizes the model's output distribution and instruction compliance behavior, while hidden-state distillation consolidates the model's multilingual internal representations. This bidirectional collaboration preserves the model's original language capabilities and task execution behavior, solving the problem of overall model capability degradation caused by low-resource adaptation and effectively suppressing catastrophic forgetting. By freezing the model backbone, routing parameters, unselected experts, and shared dense layers, only a small number of low-rank residual branches, compaction coefficients, and transmitter parameters are trained, significantly reducing training, storage, and deployment costs. This allows for rapid adaptation to more low-resource languages ​​and tasks under limited computing power, and is suitable for large-scale multilingual application scenarios. Quantitative verification can be performed through indicators such as task performance, behavior preservation, and hidden-state stability to ensure the reliability and effectiveness of the solution. At the same time, it can be flexibly extended to more language types, task families, model sizes, and continuous learning scenarios, with broad application and promotion value.

[0123] A second aspect of this application, corresponding to the method for cross-language, same-task capability transfer and behavior preservation of low-resource, multilingual, large language models provided in this application, provides a system for cross-language, same-task capability transfer and behavior preservation of low-resource, multilingual, large language models. For example... Figure 2 As shown, the low-resource, multilingual, large language model cross-lingual same-task ability transfer and behavior preservation system provided in the second embodiment of this application includes:

[0124] The data construction and encoding module 201 is used to freeze the original parameters, routing parameters, unselected expert parameters, and shared dense layer parameters of the pre-trained dynamic hybrid expert backbone model; construct language codes for each language, task codes for each task family, and language group codes for each language group; and construct the observed set O and the target transfer set P. The observed language task pairs belong to the observed set O, and the target language task pairs belong to the target transfer set P. The observed set O contains language task pairs with supervised sample data, and the target transfer set P contains language task pairs with unsupervised sample data or with a supervised sample data amount less than a preset threshold.

[0125] The high-conflict expert screening module 202 is used to calculate the conflict score of each expert branch in each hybrid expert layer of the dynamic hybrid expert backbone model based on the task gradient generated by different language task pairs being routed to the same expert branch and the corresponding routing ratio. The module then sorts the conflict scores of each expert branch from high to low and selects the top-performing experts. The expert branches corresponding to each conflict score constitute a high-conflict expert set H;

[0126] The combined low-rank parameter decoupling module 203 is used to construct shared residual branches, language group residual branches, and task-private residual branches, and to calculate the effective projection matrix for each expert branch and the projection matrix q of each expert branch in the high-conflict expert set H. The effective projection matrix is ​​obtained by linearly superimposing the frozen original projection weights, shared residual branches, language group residual branches, and task-private residual branches.

[0127] Task-private capability transmission module 204 is used to obtain the prediction task-private compact coefficient vector of the target language task pair through the task-private capability transmitter, using the language encoding, the task encoding, and the language group encoding as input. And based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task-private residual branch, wherein the regression objective of the stopping gradient during the training phase of the task-private capability transmitter is the task-private compactness coefficient corresponding to the observed language task pair. The observed language task pairs have corresponding task-private compactness coefficients. Acquired through supervised training;

[0128] Behavior retention and forgetting inhibition module 205, used for:

[0129] Using the stable model checkpoint of the dynamic hybrid expert backbone model prior to the adaptation phase as the teacher model, and the dynamic hybrid expert backbone model after the adaptation phase as the student model, a given retention set is used. Simultaneously, temperature-scaled KL divergence is used as the output-level distillation loss, serving as the input to both the teacher and student models. Output-level distillation is performed, and N layers of hybrid expert layers corresponding to the teacher model and the student model are selected as distillation alignment layers. For each distillation alignment layer, a loss based on the hidden state-level representation preservation is applied. The teacher model and the student model are aligned by distillation of the hidden state. The final sequence hidden state is the hidden state obtained after the output of the feedforward expert module in the distillation alignment layer is connected by residual connection. The retention set contains language task pairs used to constrain and evaluate the original language ability, output format, instruction compliance and response pattern stability of the model.

[0130] Calculate the subspace orthogonality regularization loss based on the subspace overlap of any two low-rank basis matrices with the same row dimension. And the task supervision loss is based on the standard causal language model supervision loss. ;

[0131] The dynamic hybrid expert backbone model is trained with the goal of minimizing the global loss, where the global loss is the output-level distillation loss. The hidden state level represents the retention loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum.

[0132] The language encoding is obtained by a language encoder through parameter deviation statistical features generated by short-step monolingual adaptation; the task encoding is a learnable task embedding vector; and the language group encoding is obtained based on the centroid vectors of language encodings under the same language family, the same writing system, or linguistic similarity.

[0133] Specifically, the high-conflict expert screening module 202, when calculating the conflict score of an expert branch based on the task gradient generated by different language task pairs being routed to the same expert branch and the corresponding routing ratio, is used to:

[0134] For each valid conflict analysis pair, compute the two language task pairs in the valid conflict analysis pair. and Conflict values ​​between task gradient directions Among them, when the language task is language task pair They are different, correspond to different task families, and and The routing ratio on the same expert is greater than or equal to a preset threshold. hour, and To form an effective conflict analysis pair, , and Representing language, and Indicates a task family. Indicates expert branch, Indicates the hybrid expert layer number. Indicates the first Expert IDs in a hybrid expert layer Representational language task pairs In the expert branch The task gradient on top, Representational language task pairs In the expert branch The task gradient is expressed as max(), which represents taking the maximum value. Vectorization is represented. Indicates the inner product. Represents the L2 norm, It is a very small positive number, used to prevent the denominator from being zero;

[0135] For each expert branch Computational expert branch The average of the conflict values ​​of all valid conflict analysis pairs is used to obtain the expert branch. Conflict score If expert branches If no corresponding valid conflict analysis pair exists, then the conflict score of the expert branch will be... Set to 0.

[0136] Specifically, the shared residual branch is used to capture cross-language general adaptation signals, the language group residual branch is used to capture adaptation signals of language family, writing system, or language type, and the task-private residual branch is used to carry the transferable task execution behavior of the task family. The task-private residual branch is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair; the combined low-rank parameter decoupling module 203 is used for:

[0137] Calculate the effective projection matrix ;

[0138] in, Let represent the expert branch, and q represent the projection matrix of the expert branch. Represents the frozen original projection weights;

[0139] Indicates a shared residual branch. , The low-rank projection matrix represents the shared residual branch. Represents the low-rank reconstruction matrix of the shared residual branches;

[0140] Represents the residual branches of language groups. , The low-rank projection matrix represents the residual branch of the language group. This represents the low-rank reconstruction matrix of the residual branches of the language group, where g is the language group index. This indicates the language group to which language u belongs; all languages ​​within the same language group share the same set of low-rank parameters.

[0141] This represents a task-private residual branch, which is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair. , and These are the first and second low-rank basis matrices shared across language groups t, respectively. Represents a diagonal function. For observed language task pairs The task-private compact coefficient vector is used to control the combination weights and activation strengths of the task behavior subspace under the combination of language u and task family t.

[0142] Specifically, the task private capability transmission module 204 is used for:

[0143] Using the concatenation result of the language encoding, the task encoding, and the language group encoding as input, the predicted task-private compact coefficient vector of the target language task pair is obtained through the task-private capability transmitter. ,in, Representation Language Language encoding, Represents task family Task coding, Representation Language The language group code of the language group to which it belongs. Indicates splicing, This refers to a task-private capability transmitter, and the training loss during the training phase of the task-private capability transmitter. , This indicates that the gradient operation is stopped. This represents the task-private compactness coefficient of the observed language task.

[0144] In obtaining the private compact coefficient vector of the prediction task for the target language task pair Then, based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task's private residual branch .

[0145] Specifically, the behavior retention and forgetting inhibition module 205 is used for:

[0146] Calculate the output stage distillation loss ,in, This indicates the output stage distillation temperature coefficient. This represents the total number of samples in the retained set. To preserve individual sample pairs in the set, Given the input sequence, For the corresponding target output sequence, Indicates the length of the token in the target output sequence. The token position index in the target output sequence, with a value range of 1. , This represents all prefix tokens preceding the j-th token in the target output sequence. Teacher model At temperature Given input and prefix The next token prediction distribution at that time Student model At temperature Given input and prefix The next token prediction distribution at that time Indicates KL divergence;

[0147] Calculate the hidden state level representation retention loss ,in, For a pre-selected set of distillation-aligned layers, The number of distillation alignment layers, , For the input sequence The length of the token. Teacher model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Student model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Denotes the square of the Frobenius norm;

[0148] Calculate total distillation loss For output stage distillation losses And hidden state level representation maintain loss The weighted sum, ,in, Distillation weights for hidden states;

[0149] Calculate the subspace overlap of any two low-rank basis matrices X and Y with the same row dimension. ,in, and The subspace orthogonal regularization loss is the matrix obtained by normalizing the column vectors of the low-rank basis matrices X and Y, respectively. The average value of the overlap of all subspaces over the current training language task pair, the selected set of high-conflict experts, and the set of projection matrices;

[0150] Calculate the monitoring loss of the task ,in, This represents the batch of current training language task pairs sampled from the new task dataset. This represents the total number of samples in the batch for the current language task training. Student model The set of trainable parameters Indicates the student model in parameters Given input and prefix Predict the next token as The probability of;

[0151] Calculate the global loss, which is the distillation loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum, global loss ,in, For the preset orthogonal regularization weights, This is the preset total distillation loss weight;

[0152] The dynamic hybrid expert backbone model is trained with the goal of minimizing the global loss.

[0153] In one possible implementation, the low-resource multilingual large language model cross-language same-task ability transfer and behavior preservation system further includes an evaluation and feedback module, wherein the evaluation and feedback module is used for:

[0154] For target language task pairs, calculate perplexity, ROUGE-L, chrF, target language consistency, and task effectiveness to assess the degree of task capability completion. If the task effectiveness is lower than a preset effectiveness threshold, readjust the parameters of the task private capability transmitter and recalculate the predicted task private compactness coefficient vector of the target language task pair.

[0155] For language task pairs in the retention set, the format compliance rate, instruction constraint satisfaction rate, response pattern consistency rate, and hidden state drift distance are calculated to evaluate the catastrophic forgetting suppression effect. If the format compliance rate is lower than a preset compliance threshold or the instruction constraint satisfaction rate is lower than a preset constraint threshold, the total distillation loss weight is increased or the retention set sampling ratio is increased.

[0156] The cross-language, same-task capability transfer and behavior preservation system for low-resource, multilingual, large language models provided in this application embodiment can be deployed as a training script, model adaptation plugin, cloud model service, edge incremental adaptation module, or integrated into the functional modules of a large language model continuous learning platform.

[0157] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be referred to the corresponding processes in the foregoing method embodiments, and therefore will not be repeated here.

[0158] The cross-language, same-task capability transfer and behavior preservation system for low-resource multilingual large language models provided in this application utilizes a task-private capability transmitter to predict the compactness coefficient of the target language task pair. This allows for task behavior instantiation under zero-sample or minimally supervised data conditions, reducing reliance on translation and manually labeled data, and lowering the data construction costs and implementation difficulties for low-resource languages, minority languages, and languages ​​with special writing systems. By modeling task execution behavior as composable private capabilities and relying on the shared foundation of the task family and the compactness coefficient of the language task pair, it enables cross-language reuse of task behavior. This efficiently transfers task knowledge and execution logic learned in high-resource languages ​​to low-resource languages, effectively improving the execution performance of low-resource tasks. Furthermore, it accurately identifies high-conflict expert branches and performs low-rank parameter decoupling only in gradient conflict regions, avoiding the inefficiency caused by full updates. Parameter perturbation alleviates parameter interference during multilingual and multitask training while improving low-resource adaptation stability. Combined with dual distillation constraints at the output level and hidden state, it stabilizes model output behavior and multilingual internal representations respectively, effectively preserving the model's original capabilities and suppressing catastrophic forgetting and overall performance degradation during low-resource adaptation. Simultaneously, by freezing the model backbone, routing parameters, unselected experts, and shared dense layers, and updating only a small number of low-rank residual branches, compaction coefficients, and transmitter parameters, it significantly reduces training, storage, and deployment overhead. This allows for rapid adaptation to various low-resource languages ​​and tasks with limited computing power, making it suitable for large-scale application scenarios. Furthermore, it can be quantitatively verified through metrics such as task performance, behavioral stability, and representation consistency. The solution demonstrates strong reliability and can be widely extended to various languages, tasks, and continuous learning scenarios, possessing significant promotional value.

[0159] It should be noted that the cross-language, same-task capability transfer and behavior preservation system for low-resource, multilingual, large language models provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of this application can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further split into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of this application are only for distinguishing the various modules or steps and are not considered as an improper limitation of this application.

[0160] A third aspect of the embodiments of this application also provides an electronic device, the electronic device comprising:

[0161] At least one processor; and,

[0162] A memory communicatively connected to at least one of the processors; wherein,

[0163] The memory stores instructions that can be executed by the processor to implement the aforementioned method for cross-language, same-task capability transfer and behavior preservation of a low-resource, multilingual, large language model.

[0164] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0166] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0167] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0168] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0169] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for cross-linguistic, same-task ability transfer and behavior preservation in a low-resource, multilingual, large language model, characterized in that, include: Freeze the original parameters, routing parameters, unselected expert parameters, and shared dense layer parameters of the pre-trained dynamic hybrid expert backbone model, and construct the language codes for each language, the task codes for each task family, and the language group codes for each language group. For each expert branch in the hybrid expert layer of the dynamic hybrid expert backbone model, based on the task gradients generated by different language task pairs being routed to the same expert branch and the corresponding routing ratios, the conflict score of the expert branch is calculated. The conflict scores of each expert branch are then sorted from high to low, and the top-performing branches are selected. The expert branches corresponding to the conflict scores constitute a set of highly conflicting experts, H. This is the default value; For each expert branch and its projection matrix q in the high-conflict expert set H, calculate the effective projection matrix, which is obtained by linearly superimposing the frozen original projection weights, shared residual branches, language group residual branches, and task-private residual branches. Using the language encoding, the task encoding, and the language group encoding as input, the predicted task-private compact coefficient vector of the target language task pair is obtained through the task-private capability transmitter. And based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task-private residual branch, wherein the regression objective of the stopping gradient during the training phase of the task-private capability transmitter is the task-private compactness coefficient corresponding to the observed language task pair. The observed language task pairs have corresponding task-private compactness coefficients. The observed language task pairs are obtained through supervised training and learning. They belong to the observed set O, and the target language task pairs belong to the target transfer set P. The observed set O contains language task pairs with supervised sample data, and the target transfer set P contains language task pairs with unsupervised sample data or with a supervised sample data amount less than a preset threshold. Using the stable model checkpoint of the dynamic hybrid expert backbone model prior to the adaptation phase as the teacher model, and the dynamic hybrid expert backbone model after the adaptation phase as the student model, a given retention set is used. Simultaneously, temperature-scaled KL divergence is used as the output-level distillation loss, serving as the input to both the teacher and student models. Output-level distillation is performed, and N layers of hybrid expert layers corresponding to the teacher model and the student model are selected as distillation alignment layers. For each distillation alignment layer, a loss based on the hidden state-level representation preservation is applied. The teacher model and the student model are aligned by distillation of the hidden state. The final sequence hidden state is the hidden state obtained after the output of the feedforward expert module in the distillation alignment layer is connected by residual connection. Calculate the subspace orthogonality regularization loss based on the subspace overlap of any two low-rank basis matrices with the same row dimension. And the task supervision loss is based on the standard causal language model supervision loss. ; The dynamic hybrid expert backbone model is trained with the goal of minimizing the global loss, where the global loss is the output-level distillation loss. The hidden state level represents the retention loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum.

2. The method for cross-lingual, same-task ability transfer and behavior preservation in a low-resource, multilingual, large language model according to claim 1, characterized in that, The language encoding is obtained by a language encoder through the parameter deviation statistical features generated by short-step monolingual adaptation. The task encoding is a learnable task embedding vector. The language group encoding is obtained based on the centroid vectors of language encodings under the same language family, the same writing system, or linguistic similarity. The calculation of the conflict score of the expert branch based on the task gradients generated by different language task pairs being routed to the same expert branch and the corresponding routing ratios includes: For each valid conflict analysis pair, compute the two language task pairs in the valid conflict analysis pair. and Conflict values ​​between task gradient directions Among them, when the language task is language task pair They are different, correspond to different task families, and and The routing ratio on the same expert is greater than or equal to a preset threshold. hour, and To form an effective conflict analysis pair, , and Representing language, and Indicates a task family. Indicates expert branch, Indicates the hybrid expert layer number. Indicates the first Expert IDs in a hybrid expert layer Representational language task pairs In the expert branch The task gradient on top, Representational language task pairs In the expert branch The task gradient is expressed as max(), which represents taking the maximum value. Vectorization is represented. Indicates the inner product. Represents the L2 norm, It is a very small positive number, used to prevent the denominator from being zero; For each expert branch Computational expert branch The average of the conflict values ​​of all valid conflict analysis pairs is used to obtain the expert branch. Conflict score If expert branches If no corresponding valid conflict analysis pair exists, then the conflict score of the expert branch will be... Set to 0.

3. The method for cross-linguistic, same-task ability transfer and behavior preservation in a low-resource, multilingual, large language model according to claim 2, is characterized in that... The shared residual branch is used to capture cross-language general adaptation signals, the language group residual branch is used to capture adaptation signals of language family, writing system or language type, and the task-private residual branch is used to carry the transferable task execution behavior of the task family. The task-private residual branch is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair. The effective projection matrix ; in, Let represent the expert branch, and q represent the projection matrix of the expert branch. Represents the frozen original projection weights; Indicates a shared residual branch. , The low-rank projection matrix represents the shared residual branch. Represents the low-rank reconstruction matrix of the shared residual branches; Represents the residual branches of language groups. , The low-rank projection matrix represents the residual branch of the language group. This represents the low-rank reconstruction matrix of the residual branches of the language group, where g is the language group index. This indicates the language group to which language u belongs; all languages ​​within the same language group share the same set of low-rank parameters. This represents a task-private residual branch, which is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair. , and These are the first and second low-rank basis matrices shared across languages ​​for task family t, respectively. Represents a diagonal function. For observed language task pairs The task's private compact coefficient vector.

4. The method for cross-linguistic, same-task ability transfer and behavior preservation in a low-resource, multilingual, large language model according to claim 3, is characterized in that... Using the concatenation result of the language encoding, the task encoding, and the language group encoding as input, the predicted task-private compact coefficient vector of the target language task pair is obtained through the task-private capability transmitter. ,in, Representation Language Language encoding, Represents task family Task coding, Representation Language The language group code of the language group to which it belongs. Indicates splicing, This refers to a task-private capability transmitter, and the training loss during the training phase of the task-private capability transmitter. , This indicates that the gradient operation is stopped. This represents the task-private compactness coefficient for the observed language task. In obtaining the private compact coefficient vector of the prediction task for the target language task pair Then, based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task-private residual branch .

5. The method for cross-linguistic, same-task ability transfer and behavior preservation in a low-resource, multilingual, large language model according to claim 4, characterized in that, Output stage distillation losses ,in, This indicates the output stage distillation temperature coefficient. This represents the total number of samples in the retained set. To preserve individual sample pairs in the set, Given the input sequence, For the corresponding target output sequence, Indicates the length of the token in the target output sequence. The token position index in the target output sequence, with a value range of 1. , This represents all prefix tokens preceding the j-th token in the target output sequence. Teacher model At temperature Given input and prefix The next token prediction distribution at that time Student model At temperature Given input and prefix The next token prediction distribution at that time Indicates KL divergence; Hidden state level representation maintains loss ,in, For a pre-selected set of distillation-aligned layers, The number of distillation alignment layers, , For the input sequence The length of the token. Teacher model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Student model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Denotes the square of the Frobenius norm; Total Distillation Loss For output stage distillation losses And hidden state level representation maintain loss The weighted sum, ,in, Distillation weights for hidden states; Subspace overlap of any two low-rank basis matrices X and Y with the same row dimension ,in, and The subspace orthogonal regularization loss is the matrix obtained by normalizing the column vectors of the low-rank basis matrices X and Y, respectively. The average value of the overlap of all subspaces over the current training language task pair, the selected set of high-conflict experts, and the set of projection matrices; Task monitoring loss ,in, This represents the batch of current training language task pairs sampled from the new task dataset. This represents the total number of samples in the batch for the current language task training. Student model The set of trainable parameters Indicates the student model in parameters Given input and prefix Predict the next token as The probability of; The global loss is the distillation loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum, global loss ,in, For the preset orthogonal regularization weights, This is the preset total distillation loss weight.

6. The method for cross-linguistic, same-task ability transfer and behavior preservation in a low-resource, multilingual, large language model according to claim 5, is characterized in that... Also includes: For a target language task pair, calculate perplexity, target language consistency, and task effectiveness to assess the degree of task capability completion. If the task effectiveness is lower than a preset effectiveness threshold, readjust the parameters of the task private capability transmitter and recalculate the predicted task private compactness coefficient vector of the target language task pair. For language task pairs in the retention set, the format compliance rate, instruction constraint satisfaction rate, response pattern consistency rate, and hidden state drift distance are calculated to evaluate the catastrophic forgetting suppression effect. If the format compliance rate is lower than a preset compliance threshold or the instruction constraint satisfaction rate is lower than a preset constraint threshold, the total distillation loss weight is increased or the retention set sampling ratio is increased.

7. A low-resource, multilingual, large language model cross-lingual same-task ability transfer and behavior preservation system, characterized in that, include: The data construction and encoding module is used to freeze the original parameters, routing parameters, unselected expert parameters, and shared dense layer parameters of the pre-trained dynamic hybrid expert backbone model, and to construct language codes for each language, task codes for each task family, and language group codes for each language group. The language codes are obtained by passing the parameter deviation statistical features generated by short-step monolingual adaptation through a language encoder. The task codes are learnable task embedding vectors. The language group codes are obtained based on the centroid vectors of language codes under the same language family, the same writing system, or linguistic similarity. The module also constructs an observed set O and a target transfer set P. The observed language task pairs belong to the observed set O, and the target language task pairs belong to the target transfer set P. The observed set O contains language task pairs with supervised sample data, and the target transfer set P contains language task pairs with unsupervised sample data or with supervised sample data less than a preset threshold. The high-conflict expert screening module is used to calculate the conflict score of each expert branch in each hybrid expert layer of the dynamic hybrid expert backbone model, based on the task gradient generated by different language task pairs being routed to the same expert branch and the corresponding routing ratio. The module then sorts the conflict scores of each expert branch from high to low and filters out the top-performing experts. The expert branches corresponding to each conflict score constitute a high-conflict expert set H; A combined low-rank parameter decoupling module is used to construct shared residual branches, language group residual branches, and task-specific residual branches, and to calculate an effective projection matrix for each expert branch and its projection matrix q in the high-conflict expert set H. The effective projection matrix is ​​obtained by linearly superimposing the frozen original projection weights, shared residual branches, language group residual branches, and task-specific residual branches. The shared residual branches are used to capture cross-language general adaptation signals, the language group residual branches are used to capture adaptation signals of language families, writing systems, or language types, and the task-specific residual branches are used to carry the transferable task execution behavior of the task family. The task-specific residual branches are obtained based on the shared low-rank basis matrix of the task family and the task-specific compact coefficient vector of the corresponding language task pair. The task-private capability transmission module is used to obtain the prediction task-private compact coefficient vector of the target language task pair by using the language encoding, the task encoding, and the language group encoding as input through the task-private capability transmitter. And based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task-private residual branch, wherein the regression objective of the stopping gradient during the training phase of the task-private capability transmitter is the task-private compactness coefficient corresponding to the observed language task pair. The observed language task pairs have corresponding task-private compactness coefficients. Acquired through supervised training; The behavior retention and forgetting inhibition module is used for: Using the stable model checkpoint of the dynamic hybrid expert backbone model prior to the adaptation phase as the teacher model, and the dynamic hybrid expert backbone model after the adaptation phase as the student model, a given retention set is used. Simultaneously, temperature-scaled KL divergence is used as the output-level distillation loss, serving as the input to both the teacher and student models. Output-level distillation is performed, and N layers of hybrid expert layers corresponding to the teacher model and the student model are selected as distillation alignment layers. For each distillation alignment layer, a loss based on the hidden state-level representation preservation is applied. The teacher model and the student model are aligned by distillation of the hidden state. The final sequence hidden state is the hidden state obtained after the output of the feedforward expert module in the distillation alignment layer is connected by residual connection. Calculate the subspace orthogonality regularization loss based on the subspace overlap of any two low-rank basis matrices with the same row dimension. And the task supervision loss is based on the standard causal language model supervision loss. ; The dynamic hybrid expert backbone model is trained with the goal of minimizing the global loss, where the global loss is the output-level distillation loss. The hidden state level represents the retention loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum.

8. The cross-lingual, same-task ability transfer and behavior preservation system for low-resource, multilingual, large language models according to claim 7, is characterized in that, The high-conflict expert screening module, when calculating the conflict score of an expert branch based on the task gradient generated by different language task pairs being routed to the same expert branch and the corresponding routing ratio, is specifically used for: For each valid conflict analysis pair, compute the two language task pairs in the valid conflict analysis pair. and Conflict values ​​between task gradient directions Among them, when the language task is language task pair They are different, correspond to different task families, and and The routing ratio on the same expert is greater than or equal to a preset threshold. hour, and To form an effective conflict analysis pair, , and Representing language, and Indicates a task family. Indicates expert branch, Indicates the hybrid expert layer number. Indicates the first Expert IDs in a hybrid expert layer Representational language task pairs In the expert branch The task gradient on top, Representational language task pairs In the expert branch The task gradient is expressed as max(), which represents taking the maximum value. Vectorization is represented. Indicates the inner product. Represents the L2 norm, It is a very small positive number, used to prevent the denominator from being zero; For each expert branch Computational expert branch The average of the conflict values ​​of all valid conflict analysis pairs is used to obtain the expert branch. Conflict score If expert branches If no corresponding valid conflict analysis pair exists, then the conflict score of the expert branch will be... Set to 0; The combined low-rank parameter decoupling module is specifically used for: Calculate the effective projection matrix ; in, Let represent the expert branch, and q represent the projection matrix of the expert branch. Represents the frozen original projection weights; Indicates a shared residual branch. , The low-rank projection matrix represents the shared residual branch. Represents the low-rank reconstruction matrix of the shared residual branches; Represents the residual branches of language groups. , The low-rank projection matrix represents the residual branch of the language group. This represents the low-rank reconstruction matrix of the residual branches of the language group, where g is the language group index. This indicates the language group to which language u belongs; all languages ​​within the same language group share the same set of low-rank parameters. This represents a task-private residual branch, which is obtained based on the low-rank basis matrix shared by the task family and the task-private compact coefficient vector of the corresponding language task pair. , and These are the first and second low-rank basis matrices shared across languages ​​for task family t, respectively. Represents a diagonal function. For observed language task pairs The task's private compact coefficient vector; The task private capability transmission module is specifically used for: Using the concatenation result of the language encoding, the task encoding, and the language group encoding as input, the predicted task-private compact coefficient vector of the target language task pair is obtained through the task-private capability transmitter. ,in, Representation Language Language encoding, Represents task family Task coding, Representation Language The language group code of the language group to which it belongs. Indicates splicing, This refers to a task-private capability transmitter, and the training loss during the training phase of the task-private capability transmitter. , This indicates that the gradient operation is stopped. This represents the task-private compactness coefficient for the observed language task. In obtaining the private compact coefficient vector of the prediction task for the target language task pair Then, based on the low-rank basis matrix shared by the task family and the prediction task-specific compact coefficient vector of the corresponding target language task pair, Calculate the task-private residual branch ; The behavior retention and forgetting inhibition module is specifically used for: Calculate the output stage distillation loss ,in, This indicates the output stage distillation temperature coefficient. This represents the total number of samples in the retained set. To preserve individual sample pairs in the set, Given the input sequence, For the corresponding target output sequence, Indicates the length of the token in the target output sequence. The token position index in the target output sequence, with a value range of 1. , This represents all prefix tokens preceding the j-th token in the target output sequence. Teacher model At temperature Given input and prefix The next token prediction distribution at that time Student model At temperature Given input and prefix The next token prediction distribution at that time Indicates KL divergence; Calculate the hidden state level representation retention loss ,in, For a pre-selected set of distillation-aligned layers, The number of distillation alignment layers, , For the input sequence The length of the token. Teacher model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Student model The l-th distillation alignment layer is paired with the input sequence. The output hidden state tensor, Denotes the square of the Frobenius norm; Calculate total distillation loss For output stage distillation losses And hidden state level representation maintain loss The weighted sum, ,in, Distillation weights for hidden states; Calculate the subspace overlap of any two low-rank basis matrices X and Y with the same row dimension. ,in, and The subspace orthogonal regularization loss is the matrix obtained by normalizing the column vectors of the low-rank basis matrices X and Y, respectively. The average value of the overlap of all subspaces over the current training language task pair, the selected set of high-conflict experts, and the set of projection matrices; Calculate the monitoring loss of the task ,in, This represents the batch of current training language task pairs sampled from the new task dataset. This represents the total number of samples in the batch for the current language task training. Student model The set of trainable parameters Indicates the student model in parameters Given input and prefix Predict the next token as The probability of; Calculate the global loss, which is the distillation loss. The subspace orthogonal regularization loss and the task supervision loss The weighted sum, global loss ,in, For the preset orthogonal regularization weights, This is the preset total distillation loss weight; The dynamic hybrid expert backbone model is trained with the goal of minimizing the global loss.

9. The cross-lingual, same-task ability transfer and behavior preservation system for low-resource, multilingual, large language models according to claim 6, characterized in that, It also includes an evaluation feedback module, which is used for: For a target language task pair, calculate perplexity, target language consistency, and task effectiveness to assess the degree of task capability completion. If the task effectiveness is lower than a preset effectiveness threshold, readjust the parameters of the task private capability transmitter and recalculate the predicted task private compactness coefficient vector of the target language task pair. For language task pairs in the retention set, the format compliance rate, instruction constraint satisfaction rate, response pattern consistency rate, and hidden state drift distance are calculated to evaluate the catastrophic forgetting suppression effect. If the format compliance rate is lower than a preset compliance threshold or the instruction constraint satisfaction rate is lower than a preset constraint threshold, the total distillation loss weight is increased or the retention set sampling ratio is increased.

10. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor to implement the cross-language, same-task capability transfer and behavior preservation method for low-resource, multilingual, large language models as described in any one of claims 1-6.

Images