A knowledge transfer method for multilingual machine translation

Abstract

The present invention provides a knowledge transfer method for multilingual machine translation, which can be applied to the fields of natural language processing and large language model technology. The method comprises: performing language perception evaluation on associated neurons in a teacher model related to multilingual machine translation tasks to obtain a language perception score set; classifying the associated neurons in the teacher model into language-general neurons and language-specific neurons according to the language perception score set, and extracting a weight submatrix from the weight matrix of the teacher model based on the classification results; based on a machine translation task in a specific language, using the weight submatrix to selectively update the parameters of a shared low-rank adaptation module and multiple language-specific low-rank adaptation modules in a student model to complete the knowledge transfer from the teacher model to the student model. The method provided by the present invention can solve technical problems such as parameter interference, knowledge forgetting, insufficient knowledge transfer efficiency, and insufficient utilization of language perception information.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing and large language models, and in particular to a knowledge transfer method for multilingual machine translation. Background Art

[0002] Natural Language Programming (NLP) refers to the use of technologies to enable computers to understand, generate, and interact with human language. Core NLP tasks include machine translation, text classification, and sentiment analysis. Large Language Models (LLMs) have demonstrated strong capabilities in NLP, particularly in multilingual machine translation (MMT), enabling translation between multiple languages.

[0003] Existing technical solutions still have the following technical problems when applying large language models for multilingual machine translation: parameter interference. During the fine-tuning process of using a single model to handle multiple language pairs, the parameter optimization goals for different language pairs may conflict with each other, resulting in parameter interference and affecting the overall translation performance; knowledge forgetting. The fine-tuning process may overwrite or damage the valuable knowledge learned by the model during the pre-training stage, including general language understanding capabilities and effective representation of specific languages; insufficient knowledge transfer efficiency. Existing knowledge transfer methods (such as standard knowledge distillation) mainly focus on the consistency of the output layer and fail to fully utilize and transfer the deeper structured knowledge implied in the teacher model parameters, especially language-specific knowledge; insufficient utilization of language perception information. Existing model adaptation technologies fail to effectively distinguish and utilize the language-general parameters and language-specific parameters within the model, limiting the optimization potential of model performance in multilingual environments. Summary of the Invention

[0004] In view of the above problems, the present invention provides a knowledge transfer method for multilingual machine translation.

[0005] According to a first aspect of the present invention, there is provided a knowledge transfer method for multilingual machine translation, comprising:

[0006] Perform language awareness evaluation on the associated neurons in the teacher model related to the multilingual machine translation task to obtain a language awareness score set;

[0007] classifying the associated neurons in the teacher model into language-general neurons and language-specific neurons according to the language-awareness score set, and extracting a weight sub-matrix from the weight matrix of the teacher model based on the classification results;

[0008] Based on the machine translation task of a specific language, the weight sub-matrix is ​​used to selectively update the parameters of the shared low-rank adaptation module and multiple language-specific low-rank adaptation modules in the student model to complete the knowledge transfer from the teacher model to the student model.

[0009] According to an embodiment of the present invention, the language perception evaluation of the associated neurons related to the multilingual machine translation task in the teacher model is performed, and the language perception score set obtained includes:

[0010] Based on the representation analysis method, the relevance score of each neural network layer in the teacher model for a specific language pair is calculated;

[0011] Based on the relevance score, determine the neural network layer related to the multilingual machine translation task from the teacher model to obtain the associated neural network layer;

[0012] Based on the preset perceptual metric method, the loss function used in the model training process is used to calculate the gradient of each associated neuron in the associated neural network layer for a specific language output;

[0013] Perform forward and back propagation of the gradient of each associated neuron output for a specific language to obtain the specific language perception score of each associated neuron;

[0014] The calculation operation of the specific language perception score is repeated to obtain a set of language perception scores of each associated neuron in all languages.

[0015] According to an embodiment of the present invention, the above-mentioned preset perceptual measurement methods include a sensitivity-based perceptual measurement method, an activation value-based perceptual measurement method, an adaptive gradient descent-based perceptual measurement method, a gradient pruning and model compression-based perceptual measurement method, a Brenner gradient-based perceptual measurement method, a Tenengrad gradient-based perceptual measurement method, a Laplace gradient-based perceptual measurement method, and a probe classifier-based perceptual measurement method.

[0016] According to an embodiment of the present invention, the classification of associated neurons in the teacher model into language-general neurons and language-specific neurons according to the language perception score set includes:

[0017] When the highest language perception score in the language perception score set of the associated neuron is lower than a preset threshold, the associated neuron is classified as a language-general neuron;

[0018] classifying the associated neuron as a language-specific neuron if the highest language perception score in the language perception score set of the associated neuron is greater than or equal to a preset threshold and the highest language perception score corresponds to a specific language;

[0019] Based on the classification results, the associative neurons in the associative neural network layer are defined as a language-universal neuron index set and a language-specific neuron index set for each specific language pair.

[0020] According to an embodiment of the present invention, extracting a weight submatrix from the weight matrix of the teacher model based on the classification result includes:

[0021] Based on the language-universal neuron index set and the dimension information of the weight matrix of the teacher model, a preset parameter extraction method is used to extract a language-universal weight submatrix compatible with the student model from the weight matrix of the teacher model;

[0022] Based on the language-specific neuron index set and the dimensional information of the weight matrix of the teacher model, a preset parameter extraction method is used to extract a language-specific weight sub-matrix compatible with the student model from the weight matrix of the teacher model.

[0023] According to an embodiment of the present invention, the above-mentioned preset parameter extraction method includes a parameter extraction method based on score Top-K selection, a parameter extraction method based on sparsification, a parameter extraction method based on gradient screening, a parameter extraction method based on linear structure, a parameter extraction method based on tree structure, and a parameter extraction method based on graph structure.

[0024] According to an embodiment of the present invention, the above-mentioned language-specific machine translation task uses a weight submatrix to selectively update the parameters of the shared low-rank adaptation module and multiple language-specific low-rank adaptation modules in the student model to complete the knowledge transfer from the teacher model to the student model, including:

[0025] Initialization of multiple language-specific low-rank adaptation modules is completed by performing singular value decomposition on the language-universal weight submatrix;

[0026] Initialization of the shared low-rank adaptation module is done by performing singular value decomposition on the language-specific weight submatrix.

[0027] The parameter update granularity is determined according to the requirements of the multilingual machine translation task or computing resources, and the parameters of the initialized multiple language-specific low-rank adaptation modules and the initialized shared low-rank adaptation module are updated based on the parameter update granularity and a preset selective fine-tuning mechanism.

[0028] According to an embodiment of the present invention, the preset selective fine-tuning mechanism includes a selective fine-tuning mechanism based on a customized adapter and a selective fine-tuning mechanism based on prefix tuning.

[0029] According to an embodiment of the present invention, the parameter update of the initialized multiple language-specific low-rank adaptation modules and the initialized shared low-rank adaptation module based on the parameter update granularity and the preset selective fine-tuning mechanism includes:

[0030] Freeze the original parameters of the student model;

[0031] When processing a specific language pair, the initialized shared low-rank adaptation module is activated, the initialized language-specific low-rank adaptation module corresponding to the specific language pair is activated, the other initialized language-specific low-rank adaptation modules are frozen, the original parameters of the student model, the parameters of the initialized shared low-rank adaptation module, and the parameters of the activated language-specific low-rank adaptation module are calculated to obtain the modified weights, and the modified weights are backpropagated to update the parameters of the initialized shared low-rank adaptation module and the activated language-specific low-rank adaptation module.

[0032] According to an embodiment of the present invention, the above-mentioned knowledge transfer method for multilingual machine translation further includes:

[0033] Replace the shared low-rank adaptation module with a parameter sharing module based on regularization term, a parameter dynamic sharing module based on gating mechanism, or a parameter sharing module based on pruning and quantization.

[0034] The above-mentioned knowledge transfer method for multilingual machine translation tasks provided by the present invention performs language perception evaluation and parameter extraction on the teacher model, and uses the extracted parameters to update the parameters of the student model, thereby realizing efficient and accurate knowledge transfer from the teacher model to the student model, and avoiding the parameter interference problem that may occur in multilingual machine translation tasks; by utilizing the shared low-rank adaptation module and the language-specific low-rank adaptation module in the student model, it can significantly improve the performance of multilingual machine translation, effectively alleviate parameter interference, and maintain a high parameter efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0036] Figure 1 2. This is an application scenario diagram of the knowledge transfer method for multilingual machine translation according to an embodiment of the present invention;

[0037] Figure 2 is a flow chart of a knowledge transfer method for multilingual machine translation according to an embodiment of the present invention;

[0038] Figure 3 is an architectural diagram of a knowledge transfer method for multilingual machine translation tasks according to an embodiment of the present invention;

[0039] Figure 4 is a structural block diagram of a knowledge transfer apparatus for multilingual machine translation according to an embodiment of the present invention;

[0040] Figure 5 is a block diagram of an electronic device suitable for implementing a knowledge transfer method for multilingual machine translation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0042] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0043] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0044] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0045] In the field of natural language processing, researchers have developed parameter-efficient fine-tuning (PEFT) techniques to enable large pre-trained models to efficiently adapt to downstream tasks (such as machine translation) while reducing computational and storage overhead. For example, low-rank adaptation (LoRA) fine-tunes the model by introducing low-rank matrices into the original model layers. Knowledge distillation (KD) is also used to transfer the knowledge of a large, high-performing "teacher" model to a smaller, more efficient "student" model, enabling the student model to achieve performance close to that of the teacher. Furthermore, research on the internal structure of large language models has revealed that specific neurons or parameters within the model exhibit distinct activation patterns when processing different languages. Some parameters tend to process universal linguistic features across languages ​​(language-universal parameters), while others may be specialized for processing characteristics specific to a particular language or language pair (language-specific parameters).

[0046] However, in the field of natural language processing technology, existing technical solutions that have at least one of the following technical problems: parameter interference, knowledge forgetting, insufficient knowledge transfer efficiency, insufficient utilization of language perception information, etc.

[0047] To address one of the existing technical issues, this paper provides a knowledge transfer method for multilingual machine translation tasks. The method includes a Language-Aware Parameters Detection and LoRA-Based Knowledge Transfer for Multilingual Machine Translation (MLAS-LoRA) algorithm. This method identifies and extracts language-aware parameters from a teacher model and utilizes a novel multi-LoRA architecture to inject this knowledge into a student model, enabling efficient and low-intrusion fine-tuning.

[0048] The knowledge transfer method provided by the present invention is further described in detail below through specific examples, specific implementation methods and specific experiments.

[0049] Figure 1 2 is an application scenario diagram of a knowledge transfer method for multilingual machine translation according to an embodiment of the present invention.

[0050] like Figure 1As shown, the application scenario 100 according to this embodiment may include natural language processing and a large language model. A network 104 is used as a medium for providing a communication link between a first terminal device 101, a second terminal device 102, a third terminal device 103, and a server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0051] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0052] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0053] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0054] It should be noted that the knowledge transfer method for multilingual machine translation provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the knowledge transfer apparatus for multilingual machine translation provided in the embodiment of the present invention can generally be set in the server 105. The knowledge transfer method for multilingual machine translation provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the knowledge transfer apparatus for multilingual machine translation provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0055] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0056] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The knowledge transfer method for multilingual machine translation according to the disclosed embodiment is described in detail.

[0057] Figure 2 FIG. 4 is a flowchart of a knowledge transfer method for multilingual machine translation according to an embodiment of the present invention.

[0058] like Figure 2 As shown, the knowledge transfer method for multilingual machine translation in this embodiment includes operations S210 to S230.

[0059] In operation S210 , language awareness evaluation is performed on associated neurons in the teacher model that are related to the multilingual machine translation task to obtain a language awareness score set.

[0060] During the language awareness evaluation process, we first determine the relevant neural network layers in the teacher model that are relevant to the multilingual machine translation task. For example, if the teacher model is a Transformer-based model, we find the Transformer layers that are most relevant to multilingual machine translation from the teacher model, and then determine the relevant neurons from these most relevant Transformer layers.

[0061] In operation S220 , associated neurons in the teacher model are classified into language-general neurons and language-specific neurons according to the language awareness score set, and a weight sub-matrix is ​​extracted from the weight matrix of the teacher model based on the classification result.

[0062] In operation S230, based on a machine translation task in a specific language, the weight sub-matrix is ​​used to selectively update the parameters of the shared low-rank adaptation module and multiple language-specific low-rank adaptation modules in the student model to complete the knowledge transfer from the teacher model to the student model.

[0063] According to an embodiment of the present invention, the above-mentioned knowledge transfer method for multilingual machine translation also includes: replacing the shared low-rank adaptation module with a parameter sharing module based on a regularization term, a parameter dynamic sharing module based on a gating mechanism, or a parameter sharing module based on pruning and quantization.

[0064] The above-mentioned knowledge transfer method for multilingual machine translation tasks provided by the present invention performs language perception evaluation and parameter extraction on the teacher model, and uses the extracted parameters to update the parameters of the student model, thereby realizing efficient and accurate knowledge transfer from the teacher model to the student model, and avoiding the parameter interference problem that may occur in multilingual machine translation tasks; by utilizing the shared low-rank adaptation module and the language-specific low-rank adaptation module in the student model, it can significantly improve the performance of multilingual machine translation, effectively alleviate parameter interference, and maintain a high parameter efficiency.

[0065] Figure 3 4 is an architectural diagram of a knowledge transfer method for multilingual machine translation tasks according to an embodiment of the present invention.

[0066] like Figure 3 As shown, the knowledge transfer method for multilingual machine translation tasks provided by the present invention mainly includes the steps of language perception evaluation and parameter extraction, as well as knowledge injection and fine-tuning of multiple language perception LoRA, among which FFN (Feed-Forward Network) is a feedforward neural network; Add+Norm is a combination of two operations: residual connection (Add) and layer normalization (Layer Normalization).

[0067] During the language perception evaluation and parameter extraction process, the perception of neurons in the teacher model for different language pairs is evaluated; neurons are classified as language-universal or language-specific based on the perception; corresponding parameter subsets are extracted from the teacher model based on the classification results; a student model architecture is constructed that includes a shared language-universal LoRA module and multiple language-specific LoRA modules; the LoRA modules are initialized using the extracted parameter subsets; and when fine-tuning for a specific language pair, only the shared universal LoRA module and the corresponding specific LoRA module are selectively updated. The parameter subsets are extracted based on the language perception scores of the neurons and take into account the dimensionality of the student model. The LoRA modules are initialized based on the singular value decomposition (SVD) of the extracted parameter subsets.

[0068] During the selective fine-tuning mechanism, when processing data of a specific language pair, the original parameters of the student model and the specific LoRA modules of the non-corresponding language pair are frozen, and only the general LoRA module and the specific LoRA module of the current language pair are activated and updated.

[0069] According to an embodiment of the present invention, the above-mentioned language perception evaluation of the associated neurons related to the multilingual machine translation task in the teacher model to obtain the language perception score set includes: calculating the relevance score of each neural network layer in the teacher model for a specific language pair based on the representation analysis method; determining the neural network layer related to the multilingual machine translation task from the teacher model based on the relevance score to obtain the associated neural network layer; based on a preset perception measurement method, calculating the gradient of each associated neuron in the associated neural network layer for a specific language output using the loss function used in the model training process; forward propagating and backpropagating the gradient of each associated neuron output for a specific language to obtain a specific language perception score for each associated neuron; repeating the calculation operation of the specific language perception score to obtain a language perception score set for each associated neuron in all languages.

[0070] The above-mentioned technical solution for obtaining the language perception score set of associated neurons in all languages ​​can accurately identify the translation sensitivity of the teacher model for different languages, thereby improving the translation performance and translation effect for specific languages. At the same time, the perception score calculation based on gradient propagation provides a quantitative basis for the adjustment of multilingual model parameters.

[0071] According to an embodiment of the present invention, the above-mentioned preset perceptual measurement methods include a sensitivity-based perceptual measurement method, an activation value-based perceptual measurement method, an adaptive gradient descent-based perceptual measurement method, a gradient pruning and model compression-based perceptual measurement method, a Brenner gradient-based perceptual measurement method, a Tenengrad gradient-based perceptual measurement method, a Laplace gradient-based perceptual measurement method, and a probe classifier-based perceptual measurement method.

[0072] Those skilled in the art may select one or more of the above-mentioned perception measurement methods, or select other neuron perception measurement methods according to actual needs.

[0073] According to an embodiment of the present invention, the above-mentioned classification of associated neurons in the teacher model into language-general neurons and language-specific neurons based on the language perception score set includes: when the highest language perception score in the language perception score set of the associated neurons is lower than a preset threshold, the associated neurons are classified as language-general neurons; when the highest language perception score in the language perception score set of the associated neurons is greater than or equal to a preset threshold and the highest language perception score corresponds to a specific language, the associated neurons are classified as language-specific neurons; based on the classification results, the associated neurons in the associated neural network layer are defined as a language-general neuron index set and a language-specific neuron index set for each specific language pair.

[0074] The above embodiment achieves physical isolation of language-general features from language-specific features through a dynamic partitioning mechanism based on preset thresholds, thereby improving the utilization rate of model parameters. At the same time, the establishment of a language-general neuron index set provides a stable carrier for cross-language knowledge transfer, accelerating the convergence speed of low-resource language translation tasks.

[0075] According to an embodiment of the present invention, the above-mentioned extraction of weight sub-matrices from the weight matrix of the teacher model based on the classification results includes: based on the language-general neuron index set and the dimensional information of the weight matrix of the teacher model, using a preset parameter extraction method to extract a language-general weight sub-matrix compatible with the student model from the weight matrix of the teacher model; based on the language-specific neuron index set and the dimensional information of the weight matrix of the teacher model, using a preset parameter extraction method to extract a language-specific weight sub-matrix compatible with the student model from the weight matrix of the teacher model.

[0076] According to an embodiment of the present invention, the above-mentioned preset parameter extraction method includes a parameter extraction method based on score Top-K selection, a parameter extraction method based on sparsification, a parameter extraction method based on gradient screening, a parameter extraction method based on linear structure, a parameter extraction method based on tree structure, and a parameter extraction method based on graph structure.

[0077] Those skilled in the art may select one or more of the above parameter extraction methods, or adopt other parameter extraction methods according to actual translation task requirements.

[0078] The above embodiment achieves efficient migration of teacher model knowledge through preset parameter extraction methods (such as Top-K, sparsification, graph structure, etc.); and uses the dynamic adaptation mechanism of the weight sub-matrix to support knowledge transfer across model architectures (CNN, Transformer, etc.).

[0079] The following is a further detailed description of the speech perception evaluation and parameter extraction process involved in the present invention through specific implementation methods.

[0080] The language perception evaluation and parameter extraction process includes teacher model related layer selection operation, neuron language perception scoring operation, neuron classification operation and parameter extraction operation.

[0081] The teacher model related layer selection operation: First, it is necessary to determine the Transformer layer in the teacher model that is most relevant to the machine translation task. A method based on representation analysis is used to calculate each layer Relative to a specific language pair Relevance score The score is calculated by computing the activation vectors of successive layers The L1 norm of the difference is used to measure it, as shown in formula (1):

[0082] (1),

[0083] Among them, N represents the total number of forward propagation, represents the first forward propagation, Indicates the During the forward propagation The activation vector of the layer neural network, Indicates the During the forward propagation The activation vector of the layer neural network, Indicates the The input vector of the forward propagation; Represents the L1 norm; the present invention selects the one with the highest score layers, of which Match the number of layers of the student model.

[0084] Among them, the neuronal language perception score operation: in the selected Within each layer, each neuron is further evaluated For a specific language pair A sensitivity-based approach (inspired by Taylor expansion) is used to define the neuron language perception score , as shown in formula (2):

[0085] (2),

[0086] in, is the loss function, It's a neuron In the layer The output, is the gradient of the loss with respect to the output of this neuron, represents the absolute value. This score approximately represents if the neuron The output of is set to zero, which is the amount by which the loss function changes. This score is calculated by performing forward and backpropagation on the seed sentences for a particular language pair.

[0087] Among them, neuron classification operation: analyze each neuron In all A set of language awareness scores for a specific language pair , as shown in formula (3):

[0088] (3),

[0089] in, Indicates the specific language pairs Perception score.

[0090] Set a threshold (For example, set it to 0.2 based on experience). If the neuron The highest language perception score Less than , then it is classified as a language-general neuron. This type of neuron captures general linguistic knowledge across languages. If the highest language perception score Greater than or equal to , and the highest score corresponds to specific language pairs , then it is classified as a language-specific neuron (for the specific language pairs ). These neurons are specialized in processing the nuances of a particular language pair and help mitigate parameter interference. Based on the classification results, a set of language-universal neuron indices is defined for each selected layer. and each specific language pair Language-specific neuron index set .

[0091] Among them, parameter extraction: considering the dimension of the weight matrix of the teacher model Usually larger than the dimension of the student model's weight matrix , the weight matrix of the teacher model needs to be Extract the submatrix compatible with the student model Using the neuron index set and language perception score obtained in the previous step, a perception-based extraction function Extract(.) is used to extract the language general submatrix and a specific submatrix for each specific language pair , where gen is the abbreviation of general and spe is the abbreviation of specific, as shown in formulas (4) and (5):

[0092] (4),

[0093] (5).

[0094] The above extraction process aims to preserve the structural integrity of the most relevant parts of the teacher model.

[0095] According to an embodiment of the present invention, the above-mentioned language-specific machine translation task uses a weight sub-matrix to selectively update the parameters of the shared low-rank adaptation module and multiple language-specific low-rank adaptation modules in the student model to complete the knowledge transfer from the teacher model to the student model, including: completing the initialization operation of multiple language-specific low-rank adaptation modules by performing a singular value decomposition operation on the language-universal weight sub-matrix; completing the initialization operation of the shared low-rank adaptation module by performing a singular value decomposition operation on the language-specific weight sub-matrix; determining the parameter update granularity according to the requirements or computing resources of the multilingual machine translation task, and updating the parameters of the initialized multiple language-specific low-rank adaptation modules and the initialized shared low-rank adaptation module based on the parameter update granularity and the preset selective fine-tuning mechanism.

[0096] According to an embodiment of the present invention, the preset selective fine-tuning mechanism includes a selective fine-tuning mechanism based on a customized adapter and a selective fine-tuning mechanism based on prefix tuning.

[0097] According to an embodiment of the present invention, the above-mentioned parameter update of the initialized multiple language-specific low-rank adaptation modules and the initialized shared low-rank adaptation module based on the parameter update granularity and the preset selective fine-tuning mechanism includes: freezing the original parameters of the student model; when processing a specific language pair, activating the initialized shared low-rank adaptation module, activating the initialized language-specific low-rank adaptation module corresponding to the specific language pair, freezing other initialized language-specific low-rank adaptation modules, calculating the original parameters of the student model, the parameters of the initialized shared low-rank adaptation module, and the parameters of the activated language-specific low-rank adaptation module to obtain modified weights, and back-propagating the modified weights to update the parameters of the initialized shared low-rank adaptation module and the activated language-specific low-rank adaptation module.

[0098] The following is a further detailed description of the knowledge injection and fine-tuning process of the multi-language perception LoRA involved in the above embodiment through specific implementation methods.

[0099] The knowledge injection and fine-tuning of multi-language-aware LoRA includes multi-LoRA architecture design, LoRA module initialization, and parameter selective fine-tuning.

[0100] Among them, multiple LoRA architecture design: This invention is based on standard LoRA and modifies the pre-training weights through low-rank decomposition BA , but it is extended, where represents the pre-training weights, represents the modified weight, Represents the first low-rank matrix in the LoRA module and the second low-rank matrix Perform matrix multiplication. In each (or selected) layer of the student model, introduce a dual LoRA structure: a shared language-universal LoRA module: ,in, The first universal low-rank matrix representing the shared language universal LoRA module, The second universal low-rank matrix represents the shared language-universal LoRA module, which is shared between all language pairs to capture and transfer common translation knowledge. Multiple language-specific LoRA modules: , Indicates the specific language pairs Language specific LoRA modules, where The first specific low-rank matrix representing the language-specific LoRA module, The second specific low-rank matrix representing the language-specific LoRA module, each module corresponds to the specific language pairs , used to capture and transfer knowledge unique to that language pair and mitigate parameter interference by isolating language-specific adjustments.

[0101] Among them, LoRA module initialization: use the sub-matrix extracted in stage 1 to initialize these LoRA modules. Perform singular value decomposition (SVD), as shown in formula (6):

[0102] (6),

[0103] in, represents a left singular matrix, which is an orthogonal matrix; represents the singular value matrix, which is a diagonal matrix; represents the transpose of a right singular matrix and is an orthogonal matrix.

[0104] Then, according to the first preset rank Initialize the general LoRA module as shown in formulas (7) and (8):

[0105] (7),

[0106] (8),

[0107] in, represents a left singular matrix, which is an orthogonal matrix; represents the singular value matrix, which is a diagonal matrix; represents the transpose of a right singular matrix and is an orthogonal matrix.

[0108] Similarly, for each extracted language-specific submatrix Perform SVD as shown in formula (9):

[0109] (9),

[0110] in, represents a left singular matrix, which is an orthogonal matrix; represents the singular value matrix, which is a diagonal matrix; represents the transpose of a right singular matrix and is an orthogonal matrix.

[0111] Then, according to the second preset rank Initialize the corresponding specific LoRA module as shown in formulas (10) and (11):

[0112] (10),

[0113] (11),

[0114] in, represents a left singular matrix, which is an orthogonal matrix; represents the singular value matrix, which is a diagonal matrix; represents the transpose of a right singular matrix and is an orthogonal matrix.

[0115] Among them, parameter selective fine-tuning: During the fine-tuning process, the original parameters of the student model are frozen When processing the specific language pairs Input data: Activate the shared language universal LoRA module Activate the language-specific LoRA module corresponding to the current specific language pair Freeze all other language-pair-specific LoRA modules. Calculate the modified weights , as shown in formula (12):

[0116] (12),

[0117] in, Represents the first specific low-rank matrix in the LoRA module and the second specific low-rank matrix Perform matrix multiplication.

[0118] During the back-propagation process, only the activated LoRA modules (i.e. This selective update strategy further reduces the number of trainable parameters and effectively prevents interference between different language pairs.

[0119] Those skilled in the art can adjust the granularity of the selective update according to task requirements or computing resources, for example, they can selectively update some parameters of certain layers instead of the entire LoRA module.

[0120] Skilled technicians can adapt other efficient parameter tuning techniques (such as adapters and prefix tuning) based on task requirements or computing resources, rather than being limited to LoRA, to achieve a similar dual-module (general + specific) structure and selective update mechanism. For example, a shared general adapter and multiple language-specific adapters can be designed.

[0121] The knowledge transfer method for multilingual machine translation provided by the present invention is further described in detail below through specific experiments.

[0122] In model selection: Gemma-2-9b-it is selected as the teacher model and Gemma-2-2b-it is selected as the student model.

[0123] Dataset: We use 14 language pairs from the WMT18 dataset (cs-en, en-cs, de-en, en-de, et-en, en-et, fi-en, en-fi, ru-en, en-ru, tr-en, en-tr, zh-en, en-zh). We randomly sample 200,000 sentence pairs for each translation direction for fine-tuning, and 1,000 sentence pairs for computing the neural language perception score.

[0124] Data processing: Use a unified translation instruction template (“Translate from [SRC] to [TGT]:”) to process parallel data.

[0125] Specific experimental steps (taking the en-de translation task as an example):

[0126] Perceptual Evaluation and Extraction: Using 1000 seed sentence pairs for en-de, the language perception score of each neuron in each relevant layer for en-de is calculated through forward and backward propagation through the teacher model (Gemma-2-9b-it). Language perception scores for the same neuron are also calculated for 13 other language pairs. Based on all scores and a threshold of λ = 0.2, the neuron is classified as "generic" or "en-de-specific" (or other specific language pairs). Based on the classification results, the general and specific submatrices are extracted from the teacher model's weights, with their dimensions matching those of the student model (Gemma-2-2b-it).

[0127] LoRA initialization: Perform SVD on the common sub-matrix to initialize the shared common LoRA module. Perform SVD on the specific sub-matrix to initialize the end-specific LoRA module (repeat this LoRA-specific extraction and initialization process for the other 13 specific language pairs).

[0128] Selective fine-tuning: The student model was fine-tuned using 200,000 sentence pairs from en-de. The fine-tuning epochs were set to 3, the batch size to 64, the AdamW optimizer with a learning rate of 1e-4, and 8 gradient accumulation steps. At each training step, the original parameters of the student model and all non-en-de-specific LoRA modules were frozen. Only the parameters of the general LoRA module and the en-de-specific LoRA module were updated.

[0129] Evaluation: The WMT18 test set and BLEU metric were used to evaluate the translation performance of the fine-tuned model on en-de and other language pairs. The results are shown in Table 1.

[0130] Table 1 shows the BLEU score comparison results of the proposed method (MLAS-LORA) and various baseline methods on 10 language pairs (including translation directions from xx to English and from English to xx) in the WMT18 dataset (the experiment is based on the Gemma-2-2b-it student model and the Gemma-2-9b-it teacher model).

[0131] Table 1 BLEU score comparison results

[0132]

[0133] As shown in Table 1, the MLAS-LORA method proposed in this invention achieves consistent and significantly better BLEU scores than all compared baseline methods across all tested language pairs and translation directions, with the average BLEU score being approximately 1.7 points higher than the strong baseline. This fully demonstrates the effectiveness and superiority of the technical solution of this invention in improving the performance of multilingual machine translation. In other words, compared to existing technologies, the method provided by this invention can significantly improve multilingual machine translation performance, effectively alleviate parameter interference, and maintain high parameter efficiency.

[0134] Based on the above-mentioned knowledge transfer method for multilingual machine translation, the present invention also provides a knowledge transfer device for multilingual machine translation. Figure 4 The device is described in detail.

[0135] Figure 4 4 is a structural block diagram of a knowledge transfer apparatus for multilingual machine translation according to an embodiment of the present invention.

[0136] like Figure 4As shown, the knowledge transfer apparatus 400 for multilingual machine translation in this embodiment includes a language perception evaluation module 410 , a neuron classification and parameter extraction module 420 , and a knowledge transfer module 430 .

[0137] The language perception evaluation module 410 is used to perform language perception evaluation on the associated neurons related to the multilingual machine translation task in the teacher model to obtain a language perception score set. In one embodiment, the language perception evaluation module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0138] The neuron classification and parameter extraction module 420 is used to classify the associated neurons in the teacher model into language-general neurons and language-specific neurons according to the language perception score set, and extract the weight sub-matrix from the weight matrix of the teacher model based on the classification results; in one embodiment, the neuron classification and parameter extraction module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0139] The knowledge transfer module 430 is used for language-specific machine translation tasks. It uses the weight submatrix to selectively update the parameters of the shared low-rank adaptation module and multiple language-specific low-rank adaptation modules in the student model to complete the knowledge transfer from the teacher model to the student model. In one embodiment, the knowledge transfer module 430 can be used to perform operation S230 described above and will not be repeated here.

[0140] According to an embodiment of the present invention, any multiple modules among the speech perception assessment module 410, the neuron classification and parameter extraction module 420, and the knowledge transfer module 430 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present invention, at least one of the speech perception assessment module 410, the neuron classification and parameter extraction module 420, and the knowledge transfer module 430 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware through any other reasonable method of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or a suitable combination of any of these. Alternatively, at least one of the speech perception evaluation module 410 , the neuron classification and parameter extraction module 420 , and the knowledge transfer module 430 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0141] Figure 5 is a block diagram of an electronic device suitable for implementing a knowledge transfer method for multilingual machine translation according to an embodiment of the present invention.

[0142] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 502 or programs loaded from a storage unit 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0143] Various programs and data required for the operation of the electronic device 500 are stored in the RAM 503. The processor 501, ROM 502, and RAM 503 are connected to each other via a bus 504. The processor 501 executes the programs in the ROM 502 and / or RAM 503 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and RAM 503. The processor 501 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0144] According to an embodiment of the present invention, electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to bus 504. Electronic device 500 may also include one or more of the following components connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or modem. Communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 510 as needed, so that computer programs read from the removable media can be installed into storage section 508 as needed.

[0145] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0146] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above, and / or one or more memories other than ROM 502 and RAM 503.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0149] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A knowledge transfer method for multilingual machine translation, characterized in that: The method comprises: Perform language awareness evaluation on the associated neurons in the teacher model related to the multilingual machine translation task to obtain a language awareness score set; classifying the associated neurons in the teacher model into language-general neurons and language-specific neurons according to the language perception score set, and extracting a weight sub-matrix from the weight matrix of the teacher model based on the classification result; Based on a machine translation task in a specific language, the weight submatrix is ​​used to selectively update the parameters of a shared low-rank adaptation module and multiple language-specific low-rank adaptation modules in the student model to complete the knowledge transfer from the teacher model to the student model.

2. The method according to claim 1, characterized in that The language awareness of the associated neurons in the teacher model related to the multilingual machine translation task is evaluated, and the language awareness score set obtained includes: Calculating the relevance score of each neural network layer in the teacher model for a specific language pair based on a representation analysis method; Based on the relevance score, determining a neural network layer related to the multilingual machine translation task from the teacher model to obtain an associated neural network layer; Based on a preset perceptual metric method, the loss function used in the model training process is used to calculate the gradient of each associated neuron in the associated neural network layer for a specific language output; Perform forward propagation and backpropagation on the gradient of each associated neuron output for a specific language to obtain a specific language perception score for each associated neuron; The calculation operation of the specific language perception score is repeated to obtain a language perception score set of each associated neuron in all languages.

3. The method according to claim 2, characterized in that The preset perceptual measurement method includes a sensitivity-based perceptual measurement method, an activation value-based perceptual measurement method, an adaptive gradient descent-based perceptual measurement method, a gradient pruning and model compression-based perceptual measurement method, a Brenner gradient-based perceptual measurement method, a Tenengrad gradient-based perceptual measurement method, a Laplace gradient-based perceptual measurement method, and a probe classifier-based perceptual measurement method.

4. The method according to claim 2, characterized in that Classifying the associated neurons in the teacher model into language-general neurons and language-specific neurons according to the language perception score set includes: When the highest language perception score in the language perception score set of the associated neuron is lower than a preset threshold, classifying the associated neuron as a language-general neuron; classifying the associated neuron as a language-specific neuron if a highest language perception score in the language perception score set of the associated neuron is greater than or equal to a preset threshold and the highest language perception score corresponds to the specific language; Based on the classification result, the associated neurons in the associated neural network layer are defined as a language-general neuron index set and a language-specific neuron index set for each of the specific language pairs.

5. The method according to claim 4, characterized in that Extracting a weight sub-matrix from the weight matrix of the teacher model based on the classification result includes: Based on the language-universal neuron index set and the dimension information of the weight matrix of the teacher model, a preset parameter extraction method is used to extract a language-universal weight submatrix compatible with the student model from the weight matrix of the teacher model; Based on the language-specific neuron index set and the dimension information of the weight matrix of the teacher model, the preset parameter extraction method is used to extract a language-specific weight sub-matrix compatible with the student model from the weight matrix of the teacher model.

6. The method according to claim 5, characterized in that The preset parameter extraction methods include a parameter extraction method based on score Top-K selection, a parameter extraction method based on sparsification, a parameter extraction method based on gradient screening, a parameter extraction method based on linear structure, a parameter extraction method based on tree structure, and a parameter extraction method based on graph structure.

7. The method according to claim 5, characterized in that Based on a machine translation task in a specific language, selectively updating parameters of a shared low-rank adaptation module and multiple language-specific low-rank adaptation modules in the student model using the weight submatrix to complete knowledge transfer from the teacher model to the student model includes: Initializing the plurality of language-specific low-rank adaptation modules by performing a singular value decomposition operation on the language-universal weight submatrix; Initializing the shared low-rank adaptation module by performing a singular value decomposition operation on the language-specific weight submatrix; The parameter update granularity is determined according to the requirements or computing resources of the multilingual machine translation task, and the parameters of the initialized multiple language-specific low-rank adaptation modules and the initialized shared low-rank adaptation module are updated based on the parameter update granularity and a preset selective fine-tuning mechanism.

8. The method according to claim 7, characterized in that The preset selective fine-tuning mechanism includes a selective fine-tuning mechanism based on a customized Adapter and a selective fine-tuning mechanism based on Prefix Tuning.

9. The method according to claim 7, characterized in that Updating parameters of the initialized multiple language-specific low-rank adaptation modules and the initialized shared low-rank adaptation module based on the parameter update granularity and the preset selective fine-tuning mechanism includes: Freezing the original parameters of the student model; When processing the specific language pair, activate the initialized shared low-rank adaptation module, activate the initialized language-specific low-rank adaptation module corresponding to the specific language pair, freeze other initialized language-specific low-rank adaptation modules, calculate the original parameters of the student model, the parameters of the initialized shared low-rank adaptation module, and the parameters of the activated language-specific low-rank adaptation module to obtain modified weights, backpropagate the modified weights, and update the parameters of the initialized shared low-rank adaptation module and the activated language-specific low-rank adaptation module.

10. The method according to claim 1, characterized in that Also includes: The shared low-rank adaptation module is replaced by a parameter sharing module based on a regularization term, a parameter dynamic sharing module based on a gating mechanism, or a parameter sharing module based on pruning and quantization.

Images