A Federated Multilingual Machine Translation Method Based on Efficient Fine-Tuning

Abstract

This invention relates to a federated multilingual machine translation method based on efficient fine-tuning, belonging to the field of natural language processing technology. Addressing the problems of high communication costs and long training times in federated learning-based multilingual machine translation methods, this invention proposes a federated multilingual machine translation method based on efficient fine-tuning, comprising: efficient fine-tuning of the client's multilingual translation model; gradient similarity clustering of the fine-tuned multilingual translation model; average aggregation of the clustered multilingual translation model; and deployment of a federated multilingual machine translation device based on efficient fine-tuning. This invention significantly reduces computational and communication overhead while maintaining translation performance, making it suitable for distributed translation tasks in multilingual scenarios.

Description

Technical Field

[0001] This invention relates to a federated multilingual machine translation method based on efficient fine-tuning, belonging to the field of natural language processing technology. Background Technology

[0002] The performance of current data-driven multilingual machine translation models is limited by the size of the training corpus. Large-scale multilingual data sharing training is an effective way to improve the performance of multilingual machine translation models. However, due to data privacy issues, corpus sharing is difficult to achieve. Multilingual machine translation based on federated learning is a new approach to achieve corpus sharing training. Multilingual machine translation technology under the federated learning framework protects user data privacy through distributed training and provides personalized translation services for clients of different languages, becoming a research hotspot in recent years. Traditional federated learning methods achieve knowledge sharing by aggregating global model parameters, but face significant challenges: the number of parameters in pre-trained multilingual models is huge, and the frequent transmission of full parameters between clients and servers leads to extremely high communication costs, making it difficult to adapt to the needs of a surge in the number of clients in real-world scenarios. In addition, the heterogeneity of client data is significant in multilingual scenarios. High-resource languages ​​(such as English and Chinese) dominate the direction of global model updates due to massive amounts of data, while low-resource languages ​​(such as Southeast Asian minorities) experience a gradual degradation in translation performance during iteration due to the scarcity of local data, forming a "Matthew effect". Existing efficient parameter fine-tuning methods (such as Adapter-Tuning and Prompt-Tuning) reduce computational overhead by freezing the backbone parameters and fine-tuning only some modules, but they still have limitations: the Adapter module needs to insert additional computation layers into the model, which leads to increased inference latency; at the same time, the parameter compression ratio of existing methods is insufficient, making it difficult to meet the stringent requirements of federated learning for communication efficiency.

[0003] To address the aforementioned issues, there is an urgent need for a federated learning framework that balances communication efficiency, computational overhead, and multilingual performance. This invention proposes a federated multilingual machine translation method based on efficient fine-tuning. Summary of the Invention

[0004] The technical problem this invention aims to solve is as follows: This invention provides a federated multilingual machine translation method based on efficient fine-tuning to alleviate the problems of high communication costs, performance degradation of low-resource languages, long training time, and high computational overhead faced by multilingual machine translation in federated learning. This invention significantly reduces the number of communication parameters by freezing the backbone parameters of the pre-trained model during local training on the client side and only fine-tuning the low-rank matrix. At the same time, it uses a gradient similarity clustering method to divide the clients, alleviating the problem of data heterogeneity among clients. Finally, it performs unbiased average aggregation on languages ​​in the same cluster to balance the contributions of high- and low-resource languages ​​to the multilingual translation model. This invention greatly reduces computational and communication overhead while maintaining translation performance, and is suitable for distributed translation tasks in multilingual scenarios.

[0005] The technical solution of the present invention is: a federated multilingual machine translation method based on efficient fine-tuning, and the method includes:

[0006] Step1. Efficient fine-tuning of the multilingual translation model on the client side: The client inserts low-rank adapter modules into the encoder and decoder of the pre-trained multilingual translation model, freezes the parameters of the backbone network, and only trains the low-rank matrices; after each round of local training, the client only uploads the fine-tuned parameters to the server;

[0007] Step2. Gradient similarity clustering of the fine-tuned multilingual translation model: On the server side, through the calculation of client gradient similarity, clients with similar language features are dynamically assigned to aggregation centers, and thus divided into several clusters;

[0008] Step3. Average aggregation of the multilingual translation model based on clustering: The average aggregation mechanism of the multilingual translation model based on clustering balances the contributions of high- and low-resource language pairs to the multilingual translation model;

[0009] Step4. Deploy the federated multilingual machine translation device based on efficient fine-tuning: Package the multilingual translation model after average aggregation in Step3 into an API interface and deploy it on the server to support concurrent requests from multiple clients, for realizing low-latency and high-concurrency multilingual translation services.

[0010] Further, the specific steps of Step1 include:

[0011] Step1.1. The client inserts low-rank adapter modules into the encoder and decoder of the multilingual translation model;

[0012] Specifically, the client obtains a local model by training local data , and only adjusts the parameters during the communication process through low-rank decomposition , where can be decomposed into two low-rank matrices A and B, and the local parameter freezing and low-rank update process are calculated as follows: ; where, BA represents the product of low-rank matrices, where , , the rank r1 << min(d, k), and d, k represent the dimensions of the current matrix;

[0013] Step1.2. At the end of each round of training, the client only transmits the updated LoRA parameters to the server, rather than transmitting the complete model parameters.

[0014] Further, the specific steps of Step2 include:

[0015] Step 2.1 First, by analyzing the changes in local model parameters, the number of cluster centers is dynamically determined;

[0016] During the initialization phase, the fine-tuned multilingual translation model first initializes the parameters. Send to all clients; at this point, the server has only one central hub, i.e., N=1.

[0017] The m-th client received Then, the local model parameters for the m-th client are trained based on its local data. And upload it to the server. Let m be the parameters uploaded to the server by client m; then what is the corresponding weight change between the m-th client and the n-th global center? The calculation is shown in the following formula: Where M is the number of clients; the local model parameters for the m-th client are obtained through training. The method is as follows:

[0018] (1) The m-th client first freezes the received data. ;

[0019] (2) Insert a low-rank adapter module into the encoder and decoder of the local model of the m-th client;

[0020] (3) The m-th client uses local data to train the low-rank matrices A and B in the low-rank adapter module. Finally, the results are obtained through the formula... The updated local model parameters for the m-th client are calculated. ;

[0021] Step 2.2: Based on the distance between the parameters uploaded to the server and the cluster center, assign the source and target languages ​​within the client to the nearest cluster center; clustering of all client languages ​​is completed using this method; calculate the distance between the weight changes of each client using cosine similarity. Its calculation is shown in the following formula:

[0022]

[0023] Step 2.3: Perform clustering to find the number of centers N;

[0024] Suppose that the given set of M clients constitutes an object set. ;

[0025] First, randomly select an object from S. Determine whether it already belongs to an existing cluster;

[0026] If it belongs to an existing cluster, select the next object from S;

[0027] If it does not belong to an existing cluster, then Consider it as the new center point and calculate its relationship with other objects. The distance between them; if and The distance is less than the set value, and If it has not yet been assigned to any existing cluster, then it will and They were all incorporated into the new cluster;

[0028] Repeat the above process until all objects in S are assigned to a cluster; through this process, an appropriate number of aggregation centers can be initialized.

[0029] Furthermore, the specific steps of Step 3 include:

[0030] Based on the segmentation results, hierarchical aggregation processing is then implemented on the server side during the multilingual translation model aggregation stage;

[0031] Specifically, in the feedforward network layer FFN, the FFN parameters of languages ​​within the same cluster are unbiasedly averaged and aggregated using a multilingual translation model averaging aggregation mechanism:

[0032] ;in, This represents the k-th cluster formed by an iterative clustering strategy based on gradient similarity, where each cluster contains a group of clients with similar language features or resource distributions. This represents the number of clients in the k-th cluster; This represents the parameters of the global feedforward network layer; This represents the parameter update amount of the q-th client on the feedforward network layer;

[0033] For the attention layer parameters, cross-cluster global average aggregation is performed:

[0034] ;in, Represents the joint set of all clusters; This represents the parameters of the global attention layer; This represents the amount of parameter updates on the attention layer;

[0035] Finally, the aggregation results from different layers are integrated into the parameters of the multilingual translation model. :

[0036] .

[0037] Furthermore, the specific steps of Step 4 include:

[0038] Step 4.1: Save the multilingual translation model after average aggregation in Step 3 as a .pth weight file, load it into the server memory using the Flask framework, and encapsulate it as a RESTful API interface; the interface supports concurrent requests in JSON format, handles multi-user translation tasks simultaneously, and integrates the function of dynamically loading model weights;

[0039] Step 4.2: Build a web front-end interactive interface based on Vue.js. After the user selects the source language and target language and enters the text to be translated, the user sends a request to the server via the HTTP protocol. After receiving the request, the server calls the preprocessing module to perform standardized word segmentation, sub-word encoding and numerical processing on the text.

[0040] Step 4.3: The preprocessed data is input into the multilingual translation model that has been averaged and aggregated in Step 3 for real-time inference. It supports multilingual translation and the translation results are returned to the front-end interface for visualization through the output module.

[0041] The present invention also provides a federated multilingual machine translation system based on efficient fine-tuning, the system comprising: a module for performing the federated multilingual machine translation method based on efficient fine-tuning.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the federated multilingual machine translation method based on efficient fine-tuning.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the federated multilingual machine translation method based on efficient fine-tuning.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the federated multilingual machine translation method based on efficient fine-tuning.

[0045] The beneficial effects of this invention are:

[0046] 1. This invention innovatively introduces low-rank adaptive technology into the federated learning framework. By decomposing the client model update quantity into a low-rank matrix product form, it achieves efficient parameter fine-tuning and reduces communication parameters by 98%.

[0047] 2. This invention addresses the heterogeneity problem among different client languages ​​in machine translation tasks. On the server side, it dynamically assigns clients with similar language features to appropriate aggregation centers through client gradient similarity calculation, thereby dividing them into several clusters, effectively mitigating the negative impact of data heterogeneity among different client languages.

[0048] 3. This invention designs a cluster-based global model averaging aggregation mechanism. Based on the gradient similarity iterative clustering strategy, clients with similar language features are divided into several sub-clusters. Finally, the model averaging is performed on each sub-cluster on the server side, thereby assigning higher aggregation weight coefficients to low-resource clients and effectively alleviating the bias problem of high-resource clients towards low-resource clients.

[0049] 4. This invention compresses the size of trainable parameters through low-rank matrix factorization and dynamically divides client clusters by combining gradient similarity clustering strategy. It performs unbiased average aggregation on the feedforward layer parameters of languages ​​in the same cluster and cross-cluster global average aggregation on the attention layer parameters. This significantly alleviates the performance degradation problem of low-resource languages ​​while reducing communication costs, and achieves dual optimization of efficiency and performance in multilingual translation tasks. This invention greatly reduces computation and communication overhead while maintaining translation performance, and is suitable for distributed translation tasks in multilingual scenarios. Attached Figure Description

[0050] Figure 1 This is a model framework diagram of the federated multilingual machine translation method based on efficient fine-tuning in this invention;

[0051] Figure 2 This is a diagram of the gradient similarity-based iterative clustering strategy in this invention;

[0052] Figure 3 This is an overall flowchart of the federated multilingual machine translation method based on efficient fine-tuning in this invention. Detailed Implementation

[0053] Example 1: As Figures 1-3 As shown, a federated multilingual machine translation method based on efficient fine-tuning is described, the method comprising:

[0054] Step 1: Efficient fine-tuning of the multilingual translation model on the client side: To address the problem of high communication overhead in federated learning in multilingual machine translation scenarios, the client side of this invention inserts a low-rank adapter module into the encoder and decoder of the pre-trained multilingual translation model, freezes the backbone network parameters, and trains only the low-rank matrix; after each round of local training, the client side only uploads the fine-tuning parameters to the server, which greatly reduces computation and communication overhead.

[0055] Step 2. Perform gradient similarity clustering on the fine-tuned multilingual translation model: To address the problem that the parameters in the model training process cannot be effectively shared due to the heterogeneity (such as vocabulary and grammar differences) between different client languages in the machine translation task, in this invention, on the server side, clients with language-similar features are dynamically assigned to aggregation centers through client gradient similarity calculation, and thus divided into several clusters, effectively alleviating the negative impact brought by data heterogeneity between different client languages;

[0056] Step 3. Perform average aggregation on the clustering-based multilingual translation model: Regarding the significant differences in the scale of translation corpora of different language clients, the traditional global average aggregation method based on data volume weighting is prone to causing the model parameters to shift towards high-resource languages. The average aggregation mechanism of the clustering-based multilingual translation model in this invention balances the contributions of high- and low-resource languages to the multilingual translation model, effectively alleviating the model parameter shift problem;

[0057] Step 4. Deploy a federated multilingual machine translation device based on efficient fine-tuning: Package the multilingual translation model after average aggregation in Step 3 as an API interface and deploy it on the server to support concurrent requests from multiple clients for implementing low-latency and high-concurrency multilingual translation services.

[0058] Furthermore, the specific steps of Step 1 include:

[0059] Step 1.1. To solve the problem of high communication cost of the multilingual translation model in federated learning, clients insert low-rank adapters (LoRA) modules into the encoder and decoder of the multilingual translation model;

[0060] Specifically, for the multilingual translation task, the parameter changes of different language pairs show a low "intrinsic rank" in the subspace of the pre-trained model. If a client obtains a local model by training local data , only adjust the parameters during the communication process through low-rank decomposition , where [[ID=*21]] can be decomposed into two low-rank matrices A and B, and the local parameter freezing and low-rank update processes are calculated as follows:

[0061] ; where BA represents the product of low-rank matrices, where , , the rank r1 << min(d, k), and d and k represent the dimensions of the current matrix;

[0062] Step 1.2. At the end of each round of training, the client only transmits the updated LoRA parameters to the server instead of the complete model parameters. This design significantly reduces the communication overhead, especially in the federated learning scenario where client devices usually have limited bandwidth and computing resources.

[0063] In Step 2, when aggregating local parameters uploaded by various clients in the multilingual translation model, the traditional global average aggregation method based on data volume weighting easily leads to a shift in model parameters towards the high-resource language due to significant differences in the resource distribution of local translation data from different language clients (e.g., high-resource languages ​​may have tens of thousands of training samples, while low-resource languages ​​may only have hundreds). To address this issue, this invention proposes an iterative clustering strategy based on gradient similarity. This strategy first dynamically assigns source and target languages ​​with similar characteristics to appropriate aggregation centers through client gradient similarity calculation, thereby dividing them into several clusters. The specific steps of Step 2 include:

[0064] Step 2.1: First, by analyzing the changes in local model parameters, the number of cluster centers is dynamically determined, thereby accurately reflecting the local data distribution. In this invention, the server-side and each client initialize the model consistently. Therefore, during the initialization phase, the fine-tuned multilingual translation model first initializes the parameters... Send to all clients; at this point, the server has only one central hub, i.e., N=1.

[0065] The m-th client received Then, the local model parameters for the m-th client are trained based on its local data. (Parameters uploaded by client m to the server), and uploaded to the server. Let m be the parameters uploaded to the server by client m; then what is the corresponding weight change between the m-th client and the n-th global center? The calculation is shown in the following formula: Where M is the number of clients; the local model parameters for the m-th client are obtained through training. The method is as follows:

[0066] (1) The m-th client first freezes the received data. ;

[0067] (2) Insert a low-rank adapter module into the encoder and decoder of the local model of the m-th client;

[0068] (3) The m-th client uses local data to train the low-rank matrices A and B in the low-rank adapter module. Finally, the results are obtained through the formula... The updated local model parameters for the m-th client are calculated. ;

[0069] Step 2.2: Based on the distance between the parameters uploaded to the server and the cluster center, assign the source and target languages ​​within the client to the nearest cluster center; clustering of all client languages ​​is completed using this method; calculate the distance between the weight changes of each client using cosine similarity. Its calculation is shown in the following formula:

[0070] ;

[0071] Step 2.3: Perform clustering to find the number of centers N;

[0072] Suppose that the given set of M clients constitutes an object set. ;

[0073] First, randomly select an object from S. Determine whether it already belongs to an existing cluster;

[0074] If it belongs to an existing cluster, select the next object from S;

[0075] If it does not belong to an existing cluster, then Consider it as the new center point and calculate its relationship with other objects. The distance between them; if and The distance is less than the set value, and If it has not yet been assigned to any existing cluster, then it will and They were all incorporated into the new cluster;

[0076] Repeat the above process until all objects in S are assigned to a cluster; through this process, an appropriate number (i.e., N) of cluster centers can be generated, laying the foundation for subsequent clustering.

[0077] Furthermore, the specific steps of Step 3 include:

[0078] Based on the segmentation results, hierarchical aggregation processing is then implemented on the server side during the multilingual translation model aggregation stage;

[0079] Specifically, in the feedforward network layer FFN, the FFN parameters of languages ​​within the same cluster are unbiasedly averaged and aggregated using a multilingual translation model averaging aggregation mechanism:

[0080] ;in, This represents the k-th cluster formed by an iterative clustering strategy based on gradient similarity, where each cluster contains a group of clients with similar language features or resource distributions. This represents the number of clients in the k-th cluster; This represents the parameters of the global feedforward network layer; This represents the parameter update amount of the q-th client on the feedforward network layer; by eliminating the impact of data magnitude differences on parameter updates, it effectively preserves the language feature knowledge of low-resource clients.

[0081] For the attention layer parameters, cross-cluster global average aggregation is performed:

[0082] ;in, Represents the joint set of all clusters; This represents the parameters of the global attention layer; This represents the amount of parameter updates at the attention layer; through cross-cluster aggregation, clients of different languages ​​share the attention weight matrix, thereby establishing a universal representation paradigm across languages;

[0083] Finally, the aggregation results from different layers are integrated into the parameters of the multilingual translation model. :

[0084] .

[0085] This strategy effectively balances the contributions of high-resource and low-resource clients to the multilingual translation model during server-side global model aggregation. In the intra-cluster aggregation phase of the FFN layer, equal weights are assigned to low-resource and high-resource clients within the same cluster using an unbiased averaging method, mitigating parameter bias caused by differences in data volume. Furthermore, in cross-cluster aggregation at the attention layer, shared common language knowledge prevents high-resource language features from excessively dominating the attention mechanism.

[0086] Furthermore, the specific steps of Step 4 include:

[0087] Step 4.1: Save the multilingual translation model after average aggregation in Step 3 as a .pth weight file, load it into the server memory using the Flask framework, and encapsulate it as a RESTful API interface; the interface supports concurrent requests in JSON format, handles multi-user translation tasks simultaneously, and integrates the function of dynamically loading model weights;

[0088] Step 4.2: Build a web front-end interactive interface based on Vue.js. After the user selects the source language (such as Chinese) and the target language (such as English) and enters the text to be translated, the user sends a request to the server via the HTTP protocol. After receiving the request, the server calls the preprocessing module to perform standardized word segmentation, sub-word encoding and numerical processing on the text.

[0089] Step 4.3: The preprocessed data is input into the multilingual translation model that has been averaged and aggregated in Step 3 for real-time inference. It supports multilingual translation such as Chinese to English and English to Russian. The translation results are returned to the front-end interface for visualization through the output module.

[0090] This invention can be used for system verification and scenario adaptation: the system's practicality can be verified in scenarios such as international conferences and cross-border tourism; users can adjust their input in real time through the interface and obtain personalized translation results; the system is based on containerized deployment to achieve cross-platform access.

[0091] The present invention also provides a federated multilingual machine translation system based on efficient fine-tuning, the system comprising: a module for performing the federated multilingual machine translation method based on efficient fine-tuning.

[0092] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the federated multilingual machine translation method based on efficient fine-tuning.

[0093] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the federated multilingual machine translation method based on efficient fine-tuning.

[0094] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the federated multilingual machine translation method based on efficient fine-tuning.

[0095] This invention compresses the size of trainable parameters through low-rank matrix factorization and dynamically divides client clusters by combining gradient similarity clustering strategy. It performs unbiased average aggregation on feedforward layer parameters of languages ​​in the same cluster and cross-cluster global average aggregation on attention layer parameters. This reduces communication costs while significantly alleviating the performance degradation problem of low-resource languages, achieving dual optimization of efficiency and performance in multilingual translation tasks.

[0096] This invention significantly reduces the number of communication parameters by freezing the backbone parameters of the pre-trained model during local training on the client side and only fine-tuning the low-rank factorization matrix. Simultaneously, it uses gradient similarity-based clustering to partition clients, mitigating data heterogeneity issues among them. Finally, it performs unbiased average aggregation on languages ​​within the same cluster to balance the contributions of high- and low-resource languages ​​to the global model. This invention greatly reduces computational and communication overhead while maintaining translation performance, making it suitable for distributed translation tasks in multilingual scenarios.

[0097] To verify the effectiveness of the proposed federated multilingual machine translation method based on efficient fine-tuning, a comparative experiment was conducted with baseline methods. All methods used the TED2020 and Europarl datasets.

[0098] Table 1 shows the comparative experimental results of the method of this invention and different fine-tuning methods on the TED2020 dataset.

[0099]

[0100] In the table, language pairs are represented using the abbreviation form of "source language-target language," and the language codes follow the ISO 639 standard. Specifically, these include: zh-en (Chinese → English), th-en (Thai → English), ar-en (Arabic → English), he-en (Hebrew → English), fi-en (Finnish → English), et-en (Estonian → English), ru-en (Russian → English), and sl-en (Slovenian → English), and the same applies below.

[0101] Table 2 shows the comparative experimental results of the method of this invention and different fine-tuning methods on the Europarl dataset.

[0102]

[0103] In the table, language pairs are represented using the abbreviation form of "source language-target language," and the language codes follow the ISO 639 standard. de-fr (German → French), nl-pl (Dutch → Polish), en-lt (English → Lithuanian), fr-nl (French → Dutch), it-sl (Italian → Slovenian), es-lv (Spanish → Latvian), pl-en (Polish → English), sl-es (Slovenian → Spanish), sl-lt (Slovenian → Lithuanian), It-de (Italian → German), lv-it (Latvian → Italian), lv-pl (Latvian → Polish), and so on.

[0104] Table 3 compares the parameter scales of the method of this invention and different fine-tuning methods.

[0105]

[0106] The experimental results are as follows: As shown in Table 1, the average BLEU score of the method of this invention on the TED2020 dataset reached 36.34, an improvement of 1.45 compared to the baseline model (Centralized-model); as shown in Table 2, the average BLEU score of this invention was 26.19, which is better than Adapter-Tuning (26.07) and an improvement of 0.39 compared to the centralized model (25.80). As shown in Table 3, the communication parameter size was reduced to 4.3M (0.7% of the original model), verifying the balance between parameter efficiency and performance.

[0107] Table 4 shows the ablation experiments designed to verify the communication overhead and deployment costs of different methods.

[0108] Table 4 shows the overall ablation experiment results.

[0109]

[0110] Experimental results show that, based on the federated learning-based multilingual machine translation model (Fedavg), the method of this invention adds a low-order adaptive fine-tuning strategy and a global model average aggregation mechanism. Although the average translation performance of the client decreases, the parameter scale of communication between the client and the server is reduced by 98%. This indicates that the method proposed in this invention can effectively reduce the problem of huge communication overhead in federated learning, while still maintaining translation performance close to full parameter fine-tuning.

[0111] Table 5-6 shows the comparative experiments designed to verify the impact of different aggregation methods on the model translation performance.

[0112] Table 5 shows the comparative experimental results of different aggregation methods on the TED2020 dataset.

[0113]

[0114] Table 6 shows the comparative experimental results of different aggregation methods on the Europarl dataset.

[0115]

[0116] Weighted average aggregation refers to the process of averaging the local model data on the server side based on the amount of data from the client, thereby updating the global model. Average aggregation mechanism refers to the method of directly arithmetic averaging the results of iterative clustering based on gradient similarity. As shown in Tables 5 and 6, on the TED2020 and Europarl public datasets, the average aggregation mechanism significantly outperforms the weighted average aggregation method in BLEU scores, especially for low-resource clients such as sl-en and fi-en. The average aggregation mechanism improves the contribution of low-resource clients to the personalized global model, avoids ignoring client-specific knowledge, effectively mitigates client bias in federated learning, and improves client translation performance.

[0117] Example 2: According to the concept of the present invention, the present invention also provides a federated multilingual machine translation device based on efficient fine-tuning, which includes the following integrated modules:

[0118] Client-side selection module: This module is responsible for interacting with the user's choices. The user selects the appropriate translation method based on their needs within the front-end graphical user interface, i.e., choosing the source and target languages.

[0119] Input Module: The input module is responsible for interacting with user input. Users enter the text to be translated into the input box in the front-end graphical user interface, then click the translate button. This sends the input text information and the corresponding language to the server. The server processes the input data and returns the translation result, for example, Chinese to English for client 1 and English to Russian for client 2.

[0120] Preprocessing Module: The preprocessing module preprocesses the text to be translated received from the backend. The text to be translated is input into the preprocessing module, where it is converted into binary data format for model input through methods such as text-to-id conversion, word segmentation, and binarization using a shared vocabulary of the translation model. First, the input sentence is segmented into words, which are then converted into sub-words using the vocabulary. These sub-words are then converted to IDs and binarized, transforming the input sentence into a data structure that the model can recognize.

[0121] Personalized Federated Learning Multilingual Neural Machine Translation Module: The translation module performs translation by calling a pre-trained translation model.

[0122] Output module: The output module is responsible for returning the translation results of the model to the graphical user page for display.

[0123] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A federated multilingual machine translation method based on efficient fine-tuning, characterized in that: The method includes: Step 1: Efficient fine-tuning of the multilingual translation model on the client side: The client inserts a low-rank adapter module into the encoder and decoder of the pre-trained multilingual translation model, freezes the backbone network parameters, and trains only the low-rank matrix; after each round of local training, the client only uploads the fine-tuning parameters to the server. Step 2: After fine-tuning, the multilingual translation model is clustered based on gradient similarity: On the server side, clients with similar language features are dynamically assigned to aggregation centers through gradient similarity calculation, thus dividing them into several clusters. Step 3: Average aggregation based on clustering multilingual translation model: The average aggregation mechanism of clustering multilingual translation model balances the contributions of high- and low-resource languages ​​to the multilingual translation model; Step 4: Deploy a federated multilingual machine translation device based on efficient fine-tuning: Encapsulate the multilingual translation model after average aggregation in Step 3 into an API interface, deploy it on a server, support concurrent requests from multiple clients, and use it to achieve low-latency, high-concurrency multilingual translation services. The specific steps of Step 3 include: Based on the segmentation results, hierarchical aggregation processing is then implemented on the server side during the multilingual translation model aggregation stage; Specifically, in the feedforward network layer FFN, the FFN parameters of languages ​​within the same cluster are unbiasedly averaged and aggregated using a multilingual translation model averaging aggregation mechanism: ;in, This represents the k-th cluster formed by an iterative clustering strategy based on gradient similarity, where each cluster contains a group of clients with similar language features or resource distributions. This represents the number of clients in the k-th cluster; This represents the parameters of the global feedforward network layer; This represents the parameter update amount of the q-th client on the feedforward network layer; For the attention layer parameters, cross-cluster global average aggregation is performed: ;in, Represents the joint set of all clusters; This represents the parameters of the global attention layer; This represents the amount of parameter updates on the attention layer; Finally, the aggregation results from different layers are integrated into the parameters of the multilingual translation model. : 。 2. The federated multilingual machine translation method based on efficient fine-tuning according to claim 1, characterized in that: The specific steps of Step 1 include: Step 1.1: The client inserts a low-rank adapter module into the encoder and decoder of the multilingual translation model; Specifically, the client obtains a local model by training local data. By using low-rank decomposition, only parameters are adjusted during communication. ,in, The decomposition into two low-rank matrices A and B, and the local parameter freezing and low-rank update process are calculated as follows: ; where, BA represents the product of low-rank matrices, where , , the rank r1 << min(d, k), and d and k represent the dimensions of the current matrix; Step 1.2: At the end of each training round, the client only transmits the updated LoRA parameters to the server, instead of transmitting the complete model parameters.

3. The federated multilingual machine translation method based on efficient fine-tuning according to claim 1, characterized in that: The specific steps of Step 2 include: Step 2.1 First, by analyzing the changes in local model parameters, the number of cluster centers is dynamically determined; During the initialization phase, the fine-tuned multilingual translation model first initializes the parameters. Send to all clients; at this point, the server has only one central hub, i.e., N=1. The m-th client received Then, the local model parameters for the m-th client are trained based on its local data. And upload it to the server. Let m be the parameters uploaded to the server by client m; then what is the corresponding weight change between the m-th client and the n-th global center? The calculation is shown in the following formula: Where M is the number of clients; the local model parameters for the m-th client are obtained through training. The method is as follows: (1) The m-th client first freezes the received data. ; (2) Insert a low-rank adapter module into the encoder and decoder of the local model of the m-th client; (3) The m-th client uses local data to train the low-rank matrices A and B in the low-rank adapter module. Finally, the results are obtained through the formula... The updated local model parameters for the m-th client are calculated. ; Step 2.2: Based on the distance between the parameters uploaded to the server and the cluster center, assign the source and target languages ​​within the client to the nearest cluster center; clustering of all client languages ​​is completed using this method; calculate the distance between the weight changes of each client using cosine similarity. Its calculation is shown in the following formula: ； Step 2.3: Perform clustering to find the number of centers N; Suppose that the given set of M clients constitutes an object set. ; First, randomly select an object from S. Determine whether it already belongs to an existing cluster; If it belongs to an existing cluster, select the next object from S; If it does not belong to an existing cluster, then Consider it as the new center point and calculate its relationship with other objects. The distance between them; if and The distance is less than the set value, and If it has not yet been assigned to any existing cluster, then it will and They were all incorporated into the new cluster; Repeat the above process until all objects in S are assigned to a cluster; through this process, an appropriate number of aggregation centers can be initialized.

4. The federated multilingual machine translation method based on efficient fine-tuning according to claim 1, characterized in that: The specific steps of Step 4 include: Step 4.1: Save the multilingual translation model after average aggregation in Step 3 as a .pth weight file, load it into the server memory using the Flask framework, and encapsulate it as a RESTful API interface; the interface supports concurrent requests in JSON format, handles multi-user translation tasks simultaneously, and integrates the function of dynamically loading model weights; Step 4.2: Build a web front-end interactive interface based on Vue.js. After the user selects the source language and target language and enters the text to be translated, the user sends a request to the server via the HTTP protocol. After receiving the request, the server calls the preprocessing module to perform standardized word segmentation, sub-word encoding and numerical processing on the text. Step 4.3: The preprocessed data is input into the multilingual translation model that has been averaged and aggregated in Step 3 for real-time inference. It supports multilingual translation and the translation results are returned to the front-end interface for visualization through the output module.

5. A federated multilingual machine translation system based on efficient fine-tuning, characterized in that: The system includes a module for performing the federated multilingual machine translation method based on efficient fine-tuning as described in any one of claims 1 to 4.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the federated multilingual machine translation method based on efficient fine-tuning as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the federated multilingual machine translation method based on efficient fine-tuning as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the federated multilingual machine translation method based on efficient fine-tuning as described in any one of claims 1 to 4.

Images