Large model personalized federal learning method and device based on automatic splitting

Abstract

The invention relates to a large model personalized federal learning method and device based on automatic splitting, and the method comprises the steps: deploying a first model layer of a target model at a client, and deploying a second model layer of the target model at a first server; respectively introducing a low-rank adapter for the first model layer and the second model layer, and cooperatively training the low-rank adapter based on the client and the first server to obtain the trained first model layer; respectively introducing a gating function for the first model layer, inputting the intermediate vector output by the trained first model layer into the corresponding gating function to obtain a model layer weight output by the gating function, and adjusting the output of the first model layer based on the model layer weight; the adjusted parameters of the first model layer are mapped to the second server, the target model deployed in the second server and the first model layer are aggregated to obtain a global model, and the problem that a high-performance model cannot be obtained through training in a resource limited scene is solved.

Description

Technical Field

[0001] The present application relates to the field of large models, and in particular to a method and device for personalized federated learning of large models based on automatic splitting. Background Art

[0002] Large models typically require significant computing power to train. Slow hardware, limited memory, and storage can significantly increase model training time or even prevent training from completing. Furthermore, resource-constrained models may not be able to access sufficiently large datasets, limiting the model's generalization capabilities and ultimate performance. Therefore, training large models in resource-constrained environments is a challenging task.

[0003] There is currently no effective solution to the problem in related technologies that high-performance models cannot be trained in resource-constrained scenarios. Summary of the Invention

[0004] Based on this, it is necessary to provide a large-model personalized federated learning method and device based on automatic splitting that can solve the problem of being unable to train high-performance models in resource-constrained scenarios to address the above technical problems.

[0005] First, this embodiment provides a large-model personalized federated learning method based on automatic splitting, including:

[0006] Invoking a first model layer of a target model on a client, and invoking a second model layer of the target model on a first server;

[0007] Introducing low-rank adapters for the first model layer and the second model layer respectively, and collaboratively training the low-rank adapters based on the client and the first server to obtain a trained first model layer;

[0008] introducing a gating function for each of the first model layers, calculating a model layer weight based on the trained output data of the first model layer by the gating function, and adjusting the output data based on the model layer weight;

[0009] The adjusted parameters of the first model layer are mapped to the second server, and the target model deployed on the second server and the first model layer are aggregated to obtain a global model.

[0010] In some embodiments, calculating the model layer weights by the gating function based on the output data of the trained first model layer, and adjusting the output data based on the model layer weights, includes:

[0011] Obtaining output data of the first model layer after training;

[0012] Inputting the output data into the corresponding gating function to obtain the model layer weight output by the gating function;

[0013] Performing weighted summation on the output data and the model layer weight to obtain corrected data;

[0014] The output data is adjusted based on the correction data.

[0015] In some embodiments, the step of collaboratively training the plurality of low-rank adapters based on the client and the first server includes:

[0016] Obtaining first computing resources required for training of each of the first model layers and second computing resources available to the client;

[0017] determining a third model layer in the first model layer according to the first computing resource and the second computing resource;

[0018] The low-rank adapters corresponding to the second model layer and the third model layer are collaboratively trained based on the client and the first server.

[0019] In some embodiments, mapping the parameters of the first model layer to the second server includes:

[0020] Determine whether the first model layer is trained on each client;

[0021] If so, the parameters of the first model layer are mapped to the second server.

[0022] In some embodiments, aggregating the target model deployed on the second server and the first model layer to obtain a global model includes:

[0023] Obtaining a corresponding client weight according to the quality parameter of each client;

[0024] Based on the client weight, adjusting the parameters of the first model layer corresponding to the client mapped to the second server;

[0025] The target model deployed on the second server and the first model layer after parameter adjustment are aggregated to obtain the global model.

[0026] In some embodiments, the client and the first server collaboratively train the plurality of low-rank adapters to obtain the trained first model layer, including:

[0027] Freezing parameters of the first model layer and the second model layer;

[0028] Training the low-rank adapter based on a preset dataset;

[0029] The trained low-rank adapter and the corresponding first model layer are combined to obtain the trained first model layer.

[0030] In some embodiments, the first model layer is the input embedding layer and the decoding output layer of the target model, and the second model layer is the intermediate layer of the target model.

[0031] Secondly, in this embodiment, a large-model personalized federated learning device based on automatic splitting is provided, including:

[0032] A calling module, configured to call the first model layer of the target model on the client, and call the second model layer of the target model on the first server;

[0033] A first training module is configured to introduce low-rank adapters into the first model layer and the second model layer respectively, and collaboratively train the low-rank adapters based on the client and the first server to obtain the trained first model layer;

[0034] A second training module is configured to introduce a gating function into each of the first model layers, calculate a model layer weight based on the output data of the trained first model layer by the gating function, and adjust the output data based on the model layer weight;

[0035] An aggregation module is used to map the parameters of the first model layer to the second server, aggregate the target model deployed on the second server and the first model layer to obtain a global model.

[0036] In a third aspect, a computer device is provided in this embodiment, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the large-model personalized federated learning method based on automatic splitting described in the first aspect is implemented.

[0037] In a fourth aspect, a computer-readable storage medium is provided in this embodiment, on which a computer program is stored. When the computer program is executed by a processor, the large-model personalized federated learning method based on automatic splitting described in the first aspect above is implemented.

[0038] In the above-mentioned large-model personalized federated learning method and device based on automatic splitting, the client and the first server collaboratively train the input layer and intermediate layer of the target model based on the low-rank adapter, fine-tune the target model to save training resources, and adjust the weight of the first model layer trained by the client through the gating function to improve the model performance, thereby generating a high-performance model through aggregation through the second server in a resource-constrained scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a diagram illustrating an application environment of a large-model personalized federated learning method based on automatic splitting in one embodiment;

[0040] Figure 2 1. A flowchart of a large-model personalized federated learning method based on automatic splitting in one embodiment;

[0041] Figure 3 A flowchart of a large-model personalized federated learning method based on automatic splitting in another embodiment;

[0042] Figure 4 This is a structural block diagram of a large-model personalized federated learning device based on automatic splitting in one embodiment;

[0043] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] The large-model personalized federated learning method based on automatic splitting provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, multiple clients 102 communicate with the first server 104 and the second server 106 through the network. The application environment of the large model personalized federated learning method based on automatic splitting can also include a data storage system, which can store the data to be processed by the first server 104 and / or the second server 106. The data storage system can be integrated on the first server 104 or placed on the second server 106. The large model is trained in combination with multiple clients 102 and the first server 104, and the trained large language model is aggregated on the second server 106 through parameter mapping. Among them, the client 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, etc. The first server 104 and the second server 106 can be implemented as independent servers or a server cluster consisting of multiple servers.

[0046] In one embodiment, Figure 2 As shown in the figure, a large model personalized federated learning method based on automatic splitting is provided, which is applied to Figure 1 Taking this as an example, the following steps are included:

[0047] Step S202: calling the first model layer of the target model on the client, and calling the second model layer of the target model on the first server.

[0048] The first model layer contains sensitive information, including but not limited to the user's original text and tags. The first model layer of the target model can be invoked on one or more clients. By training and updating the invoked first model layer on the first client, the data in the first model layer is ensured to remain locally throughout the large model training process.

[0049] The second model layer is mainly used to provide general domain knowledge support. The second model layer belongs to the general computing model in the model. The training of the second model layer does not depend on specific tasks or user data, and therefore does not involve sensitive information.

[0050] Optionally, the target model is divided into a first model layer and a second model layer according to whether the training of the model layer involves sensitive information, the first model layer involving sensitive information is set on multiple clients respectively, and the second model layer not involving sensitive information is set on the first server.

[0051] In step S204, low-rank adapters are introduced into the first model layer and the second model layer respectively, and the low-rank adapters are collaboratively trained based on the client and the first server to obtain the trained first model layer.

[0052] Among them, a low-rank adapter is introduced for each model layer in the first model layer and the second model layer. The training of the first model layer and the second model layer is achieved by collaboratively training the parameters of the low-rank adapter rather than the parameters of the entire target model. This can significantly reduce the number of parameters required for training and reduce computing and storage requirements.

[0053] Optionally, in the case of multiple clients, a low-rank adapter is collaboratively trained based on a certain client and the first server, and the client obtains a trained first model layer based on the first model layer and the low-rank adapter associated with the first model layer. The process of collaboratively training the low-rank adapter based on the new client and the first server is performed synchronously or asynchronously until the first model layers of the multiple clients are trained. The training data and / or training tasks of each client are different.

[0054] Optionally, the client loads the first model layer of the target model and trains and updates the first model layer locally (on the client), ensuring that the data remains locally throughout the process and protecting privacy. During collaborative training, the client performs inference operations based on local data to generate personalized outputs, while only transmitting processed intermediate activation values ​​(such as feature representations) to the first server, avoiding uploading any data containing sensitive information and achieving efficient and privacy-friendly collaborative computing. The server receives the intermediate activation values ​​from the client, completes further processing based on these non-sensitive data, and returns the results to the client. Since the entire communication process only involves the processed intermediate representations and the non-sensitive parts of the model, the risk of sensitive information leakage can be minimized, ensuring the security of the training process.

[0055] In step S206, a gating function is introduced for each of the first model layers. The gating function calculates the model layer weight based on the output data of the trained first model layer, and adjusts the output data based on the model layer weight.

[0056] In the case of multiple trained first model layers, each trained first model layer is treated as a LoRA expert model, and a gating function is assigned to each first model layer. Optionally, the gating function at each layer dynamically calculates a combined weight assigned to the current first model layer based on the output data of the LoRA expert model. Based on the weights of the multiple first model layers calculated by the gating function, the outputs of the multiple first model layers are weighted and summed. The resulting weighted sum is used as a correction value to adjust the output data of the LoRA expert model (the trained first model layer). The adjusted output data is then input into the next first model layer connected to the current model layer.

[0057] Step S208: Map the adjusted parameters of the first model layer to the second server, aggregate the target model and the first model layer deployed on the second server, and obtain a global model.

[0058] Optionally, the complete target model is called on the second server, and all parameters of the adjusted first model layer are mapped to corresponding positions on the second server to obtain a global model.

[0059] Optionally, in order to reduce communication overhead and memory burden, the second server aggregation stage can also only synchronize the parameters of the low-rank adapter of the first model layer, rather than the parameters of the entire first model layer, so that it is suitable for actual application scenarios with limited bandwidth and limited computing resources.

[0060] In this embodiment, a low-rank adapter is introduced for the first model layer in the client, and the first model layer adjusted by the low-rank adapter is obtained through collaborative training of the client and the first server, thereby reducing the resources required for training the first model layer. The corresponding weight is obtained according to the first model layer in the client through a gating function, and the first model layer is weighted combined. The low-rank adapter is introduced as an "expert" into the MoE architecture, so that the global model aggregated on the second server can achieve the fusion of global generalized knowledge and local personalized capabilities. Even in the case of non-independent and identically distributed (Non-IID) data, high performance can be maintained, thus solving the problem of being unable to train a high-performance model in resource-constrained scenarios.

[0061] In one embodiment, the first model layer is the input embedding layer and the decoding output layer of the target model, and the second model layer is the intermediate layer of the target model.

[0062] The input embedding layer can also be called the head model layer of the target model, and the output layer can also be called the tail model layer of the target model. These two model layers directly determine the output effect of the final task, so their calculations and data processing are completed entirely locally. In addition, these two model layers involve understanding the original input data and generating output results, and often contain sensitive information such as the user's original text and labels. Keeping it on the client can effectively prevent privacy leaks. The middle layer (Mbody) is a general computing module. The first server is only responsible for processing the middle layer of the target model. It does not depend on specific tasks or user data and therefore does not involve sensitive information.

[0063] In this embodiment, by splitting the target model into an input embedding layer, an intermediate layer, and a decoding output layer, and calling the target model's input embedding layer and decoding output layer on the client, and the intermediate layer on the first server, sensitive data is never transmitted outside the client, fundamentally eliminating the possibility of the server inferring sensitive labels or data. Furthermore, in resource-constrained situations, the client only processes the input embedding layer (Mhead) and decoding output layer (Mtail), delegating computation of the intermediate layer (Mbody) to the server, reducing the resources required for model training.

[0064] In one embodiment, multiple low-rank adapters are collaboratively trained based on the client and the first server, including: obtaining the first computing resources required for training each first model layer and the second computing resources available to the client; determining the third model layer in the first model layer based on the first computing resources and the second computing resources; and collaboratively training the low-rank adapters corresponding to the second model layer and the third model layer based on the client and the first server.

[0065] The first computing resources include, but are not limited to, one or more of the following resources: the computing time and video memory capacity required for training each first model layer. The second computing resources include, but are not limited to, one or more of the following resources: the video memory capacity available to the client, the processing power of hardware such as a processor, etc. Optionally, by selecting a third model layer to be trained in the first model layer, the first computing resources required by the client for training the third model layer are less than or equal to the second computing resources available to the client, thereby effectively avoiding training interruptions due to insufficient computing power.

[0066] Optionally, the minimum computing time required to complete the calculation of the first several first model layers under different video memory limitations is recorded to obtain the first computing resources required for training of each first model layer. The second computing resources of the client are obtained in real time, and based on the second computing resources, the cost of allocating each first model layer to each client for training is evaluated, and the optimal solution is selected through the state transition equation. Specifically, if the video memory occupancy and computing time of a layer meet the resource constraints of the client, then when training based on the low-rank adapter, the first model layer is used as the third model layer and is included in the training range of the client; otherwise, the first model layer is included in the training range of the server.

[0067] In order to further optimize the system performance, when determining the first model layer of client training based on the state transition equation, the data transmission time between the input embedding layer, the decoding output layer and the intermediate layer can also be taken into consideration, thereby reducing the communication overhead while improving the overall throughput.

[0068] In this embodiment, each client dynamically adjusts the first model layer computational tasks it undertakes based on its available second computing resources, intelligently adapting to the hardware capabilities of different clients. Compared to traditional static partitioning methods, dynamically determining the third model layer for training not only improves the accuracy and flexibility of task allocation, but also significantly enhances the efficiency and robustness of distributed training.

[0069] Furthermore, in one embodiment, mapping the parameters of the first model layer to the second server includes: determining whether the first model layer is trained on each client separately; if so, mapping the parameters of the first model layer to the second server.

[0070] In the case where each client trains a first model layer, the parameters of the third model layer trained by each client are mapped to the second server. Conversely, if at least one client does not train the third model layer, there is no need to map the parameters of the third model layer to the second server.

[0071] In this embodiment, based on the training status of the third model layer on each client, the third model layer that needs to participate in aggregation is limited, allowing the target model to achieve a precise balance between client-side personalized data and global general knowledge, thereby improving the capabilities of the global model.

[0072] In one embodiment, multiple low-rank adapters are collaboratively trained based on the client and the first server to obtain a trained first model layer, including: freezing the parameters of the first model layer and the second model layer; training the low-rank adapter based on a preset data set; and combining the trained low-rank adapter and the corresponding first model layer to obtain the trained first model layer.

[0073] During training, the first model layer keeps its parameters frozen, enabling adjustments to its local parameters (low-rank adapters), improving training efficiency and reducing training resources. The second model layer keeps its parameters frozen and serves only as a bridge for knowledge transfer, limiting client-server interaction to the transfer of intermediate activation values ​​and sharing of non-sensitive model components, thus ensuring data privacy to the greatest extent possible.

[0074] Optionally, during collaborative inference, the client sends intermediate results (such as model activations) generated by the first model layer to the server. The server then further processes these results using the frozen second model layer and returns the results to the client, which then generates the final output. By rationally partitioning the model structure, the client's computational burden is effectively reduced while also ensuring that sensitive data is not leaked throughout the entire process. This enables efficient and privacy-preserving model fine-tuning, improving data security.

[0075] In this embodiment, the parameters of the first model layer on the client side are kept frozen, which can reduce the resource consumption required for training; the model middle layer (Mbody) on the server side is kept frozen, which can significantly reduce the amount of training calculations, speed up training, and save energy.

[0076] In one embodiment, the gating function calculates the model layer weight based on the output data of the first model layer after training, and adjusts the output data based on the model layer weight, including: obtaining the output data of the first model layer after training; inputting the output data into the corresponding gating function to obtain the model layer weight output by the gating function; weighted summing the output data and the model layer weight to obtain corrected data; and adjusting the output data based on the corrected data.

[0077] The first model layer after training the low-rank adapter can include multiple independently fine-tuned LoRAa matrices. These multiple independently fine-tuned LoRA matrices in each layer are considered a LoRA expert, and a learnable gating function is introduced at each layer. The gating function receives the outputs of multiple "LoRA experts" (independently fine-tuned LoRA matrices) in the same layer and dynamically calculates the combined weights assigned to each fine-tuning module in that layer based on the current input. The gating function multiplies the output of each "LoRA expert" by its corresponding model layer weight and accumulates the sum to generate fused corrected data. This corrected data is then added to the original output data to form the adjusted output data.

[0078] Optionally, the output data of each first model layer is input into a gating function introduced in that layer. Based on the gating function, the received output data can be concatenated and normalized to obtain a first feature representation. The gating function then generates a scalar value that matches the number of low-rank adapters based on the first feature representation, and the scalar value is adjusted based on a temperature parameter. The temperature parameter is an adjustable value. When a lower temperature parameter is obtained through adjustment, the model layer weights tend to be concentrated on a small number of high-performing LoRA experts. When a lower temperature parameter is obtained through over-adjustment, the model layer weight distribution is more uniform. The introduction of the temperature parameter allows the model to dynamically adjust the degree of concentration of the weight distribution.

[0079] In this embodiment, by combining the dynamic gating mechanism of expert mixture with the low-rank adapter (LoRA) fine-tuning technology, the client can autonomously train and adjust the local model to adapt to the local data distribution, thereby obtaining an aggregation model with excellent performance.

[0080] In one embodiment, the target model deployed on the second server and the first model layer are aggregated to obtain a global model, including: obtaining corresponding client weights according to the quality parameters of each client; adjusting the parameters of the first model layer of the second server mapped to the corresponding client based on the client weights; and aggregating the target model deployed on the second server and the first model layer after adjusting the parameters to obtain a global model.

[0081] The quality parameters of each client include at least one of the following: the amount of data used by the client for model training, the quality of the client's model updates, and the frequency of the client's model participation. The higher the data volume, the higher the client weight; conversely, the lower the client weight. The higher the update quality, the higher the client weight; conversely, the lower the client weight. The higher the participation frequency, the higher the client weight; conversely, the lower the client weight. Optionally, for layers updated by multiple clients, a weighted average is performed based on the dynamic weights to adjust the parameters of the first model layer mapped to the second server.

[0082] Furthermore, in order to limit the variation range of the global model parameters, regularization processing may be performed on the parameters of the first model layer mapped to the second server.

[0083] Furthermore, in order to ensure the compatibility between the parameters of each layer, the aggregated global model can also be checked for consistency.

[0084] In this embodiment, the parameters of the first model layer participating in the mapping are weighted averaged according to the quality parameters of the client, which significantly improves the robustness and performance of the distributed training system.

[0085] Among related technologies, the main technical routes for solving privacy protection and personalized fine-tuning of large-scale language models include: Federated Learning (FL), Split Learning (SL), and Split Federated Learning (SFL).

[0086] Federated learning achieves a certain degree of privacy protection by training models locally on each client and only uploading model parameters or gradients for aggregation, avoiding the centralized transmission of raw data. However, it has the following disadvantages: clients must fully load and train large models, which consumes significant computational and memory resources and makes deployment on edge devices difficult; it lacks the ability to handle heterogeneous client data (non-IID), which can easily lead to overfitting of models for general tasks and insufficient personalization capabilities; and it has high communication costs, which are particularly problematic when transmitting large model parameters.

[0087] Split learning divides the model into the client's first half and the server's second half, reducing the burden on the client by processing only the initial model layers. It also hides data features, protecting privacy to a certain extent. However, it has the following disadvantages: the server still needs to access the model's tail output or labels, which still poses inference risks; the split locations are fixed, not considering the heterogeneity of client devices; and it lacks personalized modeling capabilities.

[0088] Split federated learning combines the advantages of both approaches, splitting the model between the client and server and aggregating parameters in a federated manner. However, it suffers from the following drawbacks: It still uses a "head on the client, tail on the server" split approach, making label information easily inferred by the server; the model is statically segmented and unsuitable for client resource differences; and it lacks a personalized learning mechanism, resulting in poor generalization performance for specific tasks.

[0089] In the collaborative training of large language models by multiple parties, directly sharing model or label information may leak sensitive data, especially in highly private application scenarios such as healthcare and finance, leading to data privacy risks. Furthermore, due to the significant disparity in computing resources among participating parties (clients), traditional fixed model partitioning methods can overload some devices, resulting in low training efficiency or even inoperability. Furthermore, traditional federated fine-tuning strategies tend to share global model parameters, making it difficult to balance personalized performance with general capabilities, resulting in insufficient domain adaptability and performance degradation on specific domain tasks. In one embodiment, a novel federated fine-tuning method, FedDSLora (a personalized federated learning method for large models based on automatic splitting), is proposed. FedDSLora is designed for efficient, secure, and personalized fine-tuning of large language models in resource-constrained heterogeneous environments. By improving model partitioning, integrating a personalized federated learning strategy, and dynamically allocating computing load, it balances training efficiency, privacy protection, and model performance, providing a novel solution for building private, customized language models. Figure 3 This is a flowchart of a large-model personalized federated learning method based on automatic splitting in one embodiment of the present invention.

[0090] like Figure 3 As shown, the target model is first segmented into head and tail based on privacy protection. In this embodiment, a cloud server is used as the first server, and a local server is used as the second server. The client is responsible for processing the "head" and "tail" of the model, namely the input embedding layer (Mhead) and the decoding output layer (Mtail). The cloud server is only responsible for processing the middle layer (Mbody) of the model.

[0091] Taking into account the different computing resources available to different clients, the first model layer trained by each client can be dynamically allocated. Optionally, the resource evaluation module of the cloud server is used to obtain the second computing resources available to the client in real time. The resource evaluation module monitors the key hardware indicators of the client, such as the amount of video memory and the processing power of the computing unit, and calculates the resource overhead required for each layer of the model based on the following resources, including video memory occupancy and computational complexity, and matches these data with the actual resource status of the client to determine the maximum number of processable layers of the client. The distribution of computing tasks between the client and the server in the distributed machine learning system is optimized through the model layer partitioning module based on the dynamic programming algorithm. The core goal of the model layer partitioning module is to dynamically determine the training plan for the first model layer based on the computing time and video memory occupancy of each layer of the model, combined with the second computing resource limitations of the client (such as video memory capacity and maximum computing time), so as to maximize the training throughput and ensure efficient use of resources.

[0092] Optionally, during the implementation process, the model layer partitioning module first constructs a two-dimensional state table, and uses a two-bit record table to record the minimum computing time required to complete the calculation of the first several layers under different video memory limitations. For each layer, the module evaluates the cost of assigning it to the client or server, and selects the optimal solution through the state transition equation. Through the automatic splitting algorithm, the model layer partitioning module can intelligently adapt to the hardware capabilities of different clients, flexibly adjust the partitioning strategy of the head and tail layers when resources are limited, and ensure that each client processes the appropriate part according to its actual resources. This mechanism can flexibly allocate computing load according to the resource status of different devices, effectively avoid training interruptions caused by insufficient computing power, and improve the robustness and participation of the overall system. Compared with the traditional static partitioning method, the model layer partitioning module not only improves the accuracy and flexibility of task allocation, but also significantly improves the efficiency and robustness of distributed training, providing reliable technical support for heterogeneous devices to collaboratively participate in model training.

[0093] The training process of FedDSLora's domain-adaptive MoE federated learning is divided into two stages: the first stage is used to independently train the LoRA expert module, and the second stage is used to train the gating function.

[0094] In the first stage, LoRA expert modules are assigned to the first model layer of each client and the second model layer of the first server. Each LoRA expert module corresponds to a low-rank adapter. For each specific task or field, the LoRA expert module of each client is trained using the relevant dataset. Figure 3 The trainable model parameters in the first model layer loaded by the client are the model header M hi 、Model tail M ti , the second model layer (block model) M loaded by the first server c belong Figure 3 The model parameters are frozen in , ensuring that each module focuses on learning specific capabilities; where i is a positive integer corresponding to the client number. Because each client trains its own LoRA adapter locally for personalized updates based on local data, this process is completed entirely within the client and does not affect the parameters of the second model layer in the first server.

[0095] In the second phase, the client side loads the pre-trained first model layer and the LoRA expert modules trained in the first phase, freezing their parameters. A gating function is introduced in the first model layer to dynamically combine multiple LoRA expert modules. Optionally, a Mixture of Experts (MoE) mechanism is implemented on the client side to coordinate the outputs of multiple LoRA models. At the client level, where the head and tail models are deployed, a learnable gating function is introduced. Its core function is to calculate the contribution weights of different LoRA expert modules in real time based on the characteristics of the current input data. Ultimately, this weighted fusion correction is applied to the original output data of the pre-trained first model layer. This process preserves the generalization capabilities of the first model layer while enhancing the model's adaptability to specific tasks through the flexible combination of LoRA expert modules.

[0096] The core process of the gating function for weight adjustment includes input processing, parameter mapping, weight generation, and output fusion. The gating function receives a set of output data from the first trained model layer. These output data are typically in tensor form, with dimensions determined by the input sequence length and feature dimension. The gating function concatenates the outputs of all LoRA expert modules to generate a unified high-dimensional feature representation. To improve training stability, the concatenated features can also be normalized after the concatenation operation. The gating function introduces a learnable parameter matrix to map the high-dimensional feature representation to a low-dimensional space, generating a set of scalar values ​​that match the number of LoRA expert modules. In the gating function, after adjusting the scalar values ​​using the temperature parameter, the Softmax function converts them into normalized weights for each LoRA expert module, resulting in the model layer weights. Optionally, the gating function introduces a learnable parameter matrix e to perform a dot product operation on the high-dimensional feature representation. Based on the high-dimensional feature representation mapping, the relevance score vector ε corresponding to each LoRA expert module (the scalar value mentioned above) is obtained. A Softmax function with a learnable temperature coefficient τ is used to convert scalar values ​​into probability distributions, forming the client weights for each LoRA expert module. The core parameters of the gating function include the parameter matrix e and the temperature coefficient τ, which together determine how the combination of LoRA expert modules is dynamically adjusted based on the input data to adapt to different task requirements. The gating function multiplies the output data of each LoRA expert module with the corresponding model layer weights and accumulates the accumulated data to obtain a fused correction. This correction is then added to the original output data of the first model layer to form the module's final output. The gating function is trained using a parameter freezing strategy, optimizing only the parameters of the gating function (such as the parameter matrix e and the temperature coefficient τ), while keeping the parameters of the first model layer and the LoRA expert module fixed. This reduces training overhead while effectively preserving the original characteristics of the LoRA expert modules. Optionally, during training, to prevent the gating function from over-relying on a single LoRA expert module, a gated balance loss can be used to constrain the entropy of the weight distribution. This ensures that the contributions of each LoRA expert module are evenly distributed across the first model layer during training, thus preventing the loss of characteristics of other modules due to weight concentration. This allows the optimization of the gating function to take into account both the rationality of weight distribution and the improvement of task performance.

[0097] After completing domain-adaptive MoE federated learning, global model aggregation is achieved on the local server. This involves each client generating a corresponding LoRA adapter (Ai) based on the local training results, and then aggregating these low-rank adapters on the server side using the FedAvg algorithm. This aggregation process only involves updating the LoRA matrix and does not involve the transmission of all model parameters, significantly reducing communication overhead. Through this lightweight parameter sharing approach, local personalized updates from clients are globally integrated while protecting privacy, ensuring the model's generalization and synergy.

[0098] Because different clients may only be responsible for computing certain layers of the target model, the parameters they upload often differ in dimension, making traditional weighted averaging aggregation difficult. During the aggregation process, the aggregation module can address model size inconsistencies caused by varying client device resources in distributed machine learning systems. Alternatively, the local server maintains a complete target model and maps the parameters within the model layers responsible for the client to the corresponding locations in the global model. Layers not updated by the client are padded based on the initial parameters of the target model to ensure the integrity of the aggregated global model. Furthermore, to reflect each client's actual contribution to the global model, the aggregation module can also incorporate a dynamic weight adjustment mechanism. This dynamically calculates each client's weight based on quality parameters such as the client's data volume, update quality, and participation frequency. The aggregation module then performs a weighted average of the parameters for each layer in the global model. Layers not updated by any client retain their original parameters. Layers updated by multiple clients are weighted averaged based on dynamic weights to mitigate the impact of low-quality updates on the global model. After parameter mapping and weight adjustment, the module can also incorporate regularization to limit the fluctuation of global model parameters, prevent parameter drift, and improve model stability. Finally, the aggregation module can also perform consistency checks on the global model to ensure the compatibility of parameters between each layer.

[0099] Compared to the traditional FedAvg algorithm, the aggregation module achieves efficient and fair global model updates through hierarchical parameter mapping, dynamic weight adjustment, and hierarchical weighted averaging. It can flexibly handle heterogeneous model scenarios and significantly improve the robustness and performance of distributed training systems. It not only solves the aggregation problem caused by model size differences, but also ensures global model fairness and consistency through dynamic weight adjustment and regularization techniques, providing reliable technical support for the efficient operation of large-scale distributed machine learning systems.

[0100] In this embodiment, the model's input embedding layer (Mhead) and decoding generation header (Mtail) are innovatively placed on the client side, and only the intermediate activation value without sensitive information is transmitted, which reduces the risk of sensitive data leakage, improves the data security level, and fundamentally eliminates the possibility of the server inferring sensitive labels or data.

[0101] To address the heterogeneity of client device resources participating in federated training, model-level training is dynamically allocated based on computing resources, maximizing the utilization of each device's existing resources. This effectively reduces computing resource waste, avoids performance bottlenecks, and enables resource-constrained devices to participate in the federated fine-tuning of large-scale language models, ensuring overall collaborative efficiency. During training, the server-side model's intermediate layer (Mbody) remains frozen, significantly reducing the amount of training computation, accelerating training, and saving energy.

[0102] This embodiment also combines a Mixture of Experts dynamic gating mechanism with Low-Rank Adapter (LoRA) fine-tuning technology, allowing clients to autonomously train and adjust local models to adapt to the unique local data distribution. Furthermore, by dynamically aggregating low-rank adapters, the model achieves a precise balance between client-specific data and global general knowledge, improving the model's adaptability to domain-specific tasks and surpassing the personalized support capabilities of traditional federated learning. Furthermore, traditional federated learning typically requires frequent transmission of full parameters during model-level training and aggregation, while this framework only shares LoRA updates (parameter updates of the low-rank matrix), significantly reducing network bandwidth requirements and communication overhead compared to full model parameter transmission.

[0103] It should be understood that although the steps in the flowcharts involved in the various embodiments described above are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. For example, when the client has sufficient computing resources available, dynamic layer allocation may not be performed, and each client may train the first model layer. Multiple clients can perform domain-adaptive MoE federated learning synchronously or asynchronously.

[0104] Based on the same inventive concept, the embodiments of the present application also provide a large-model personalized federated learning device based on automatic splitting for implementing the large-model personalized federated learning method based on automatic splitting involved above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more embodiments of the large-model personalized federated learning device based on automatic splitting provided below can be found in the above limitations of the large-model personalized federated learning method based on automatic splitting, and will not be repeated here.

[0105] In one embodiment, Figure 4 As shown, a large-model personalized federated learning device based on automatic splitting is provided, including: a calling module, a first training module, a second training module and an aggregation module.

[0106] A calling module, configured to call the first model layer of the target model on the client, and call the second model layer of the target model on the first server;

[0107] A first training module is configured to introduce a gating function into each of the first model layers, calculate the model layer weights based on the output data of the trained first model layer by the gating function, and adjust the output data based on the model layer weights;

[0108] The second training module is used to introduce a gating function into the first model layer respectively, and input the intermediate vector output by the trained first model layer into the corresponding gating function to obtain the model layer weight output by the gating function, and adjust the parameters of the first model layer based on the model layer weight;

[0109] The aggregation module is used to map the parameters of the first model layer to the second server, aggregate the target model deployed on the second server and the first model layer to obtain a global model.

[0110] Optionally, the first model layer is the input embedding layer and decoding output layer of the target model, and the second model layer is the intermediate layer of the target model.

[0111] In one embodiment, the second training module calculates the model layer weight based on the output data of the first model layer after training by the gating function, and adjusts the output data based on the model layer weight, including: obtaining the output data of the first model layer after training; inputting the output data into the corresponding gating function to obtain the model layer weight output by the gating function; weighted summing the output data and the model layer weight to obtain corrected data; and adjusting the output data based on the corrected data.

[0112] In one embodiment, the first training module collaboratively trains multiple low-rank adapters based on the client and the first server, including: obtaining the first computing resources required for training each first model layer and the second computing resources available to the client; determining the third model layer in the first model layer based on the first computing resources and the second computing resources; and collaboratively training the low-rank adapters corresponding to the second model layer and the third model layer based on the client and the first server.

[0113] Optionally, the aggregation module maps the parameters of the first model layer to the second server, including: determining whether the first model layer is trained on each client separately; if so, mapping the parameters of the first model layer to the second server.

[0114] In one embodiment, the aggregation module aggregates the target model deployed on the second server and the first model layer to obtain a global model, including: obtaining corresponding client weights according to the quality parameters of each client; adjusting the parameters of the first model layer of the second server mapped to the corresponding client based on the client weight; aggregating the target model deployed on the second server and the first model layer after adjusting the parameters to obtain a global model.

[0115] In one embodiment, the first training module collaboratively trains multiple low-rank adapters based on the client and the first server to obtain a trained first model layer, including: freezing the parameters of the first model layer and the second model layer; training the low-rank adapter based on a preset data set; combining the trained low-rank adapter and the corresponding first model layer to obtain the trained first model layer.

[0116] Each module in the aforementioned large-model personalized federated learning device based on automatic splitting can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0117] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store large model data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a large model personalized federated learning method based on automatic splitting is implemented.

[0118] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0119] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0120] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0121] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0122] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0123] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A large-model personalized federated learning method based on automatic splitting, characterized by: The method comprises: Invoking a first model layer of a target model on a client, and invoking a second model layer of the target model on a first server; Introducing low-rank adapters for the first model layer and the second model layer respectively, and collaboratively training the low-rank adapters based on the client and the first server to obtain a trained first model layer; introducing a gating function for each of the first model layers, calculating a model layer weight based on the trained output data of the first model layer by the gating function, and adjusting the output data based on the model layer weight; The adjusted parameters of the first model layer are mapped to the second server, and the target model deployed on the second server and the first model layer are aggregated to obtain a global model.

2. The method according to claim 1, characterized in that Calculating a model layer weight by the gating function based on the output data of the trained first model layer, and adjusting the output data based on the model layer weight, including: Obtaining output data of the first model layer after training; Inputting the output data into the corresponding gating function to obtain the model layer weight output by the gating function; Performing weighted summation on the output data and the model layer weight to obtain corrected data; The output data is adjusted based on the correction data.

3. The method according to claim 1, characterized in that Collaboratively training a plurality of the low-rank adapters based on the client and the first server includes: Obtaining first computing resources required for training of each of the first model layers and second computing resources available to the client; determining a third model layer in the first model layer according to the first computing resource and the second computing resource; The low-rank adapters corresponding to the second model layer and the third model layer are collaboratively trained based on the client and the first server.

4. The method according to claim 2, characterized in that Mapping the parameters of the first model layer to the second server includes: Determine whether the first model layer is trained on each client; If so, the parameters of the first model layer are mapped to the second server.

5. The method according to claim 1, wherein Aggregating the target model of the second server and the first model layer to obtain a global model includes: Obtaining a corresponding client weight according to the quality parameter of each client; Based on the client weight, adjusting the parameters of the first model layer corresponding to the client mapped to the second server; The target model deployed on the second server and the first model layer after parameter adjustment are aggregated to obtain the global model.

6. The method according to claim 1, wherein The client and the first server collaboratively train the plurality of low-rank adapters to obtain the trained first model layer, including: Freezing parameters of the first model layer and the second model layer; Training the low-rank adapter based on a preset dataset; The trained low-rank adapter and the corresponding first model layer are combined to obtain the trained first model layer.

7. The method according to any one of claims 1 to 6, characterized in that The first model layer is the input embedding layer and decoding output layer of the target model, and the second model layer is the intermediate layer of the target model.

8. A large-model personalized federated learning device based on automatic splitting, characterized by: The device comprises: A calling module, configured to call the first model layer of the target model on the client, and call the second model layer of the target model on the first server; A first training module is configured to introduce low-rank adapters into the first model layer and the second model layer respectively, and collaboratively train the low-rank adapters based on the client and the first server to obtain the trained first model layer; A second training module is configured to introduce a gating function into each of the first model layers, calculate a model layer weight based on the output data of the trained first model layer by the gating function, and adjust the output data based on the model layer weight; An aggregation module is used to map the parameters of the first model layer to the second server, aggregate the target model deployed on the second server and the first model layer to obtain a global model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images