Big model optimization method and device based on federated learning, equipment and medium
By employing a selective aggregation mechanism and a low-rank matrix separation strategy, the communication overhead and data heterogeneity issues of large models in federated learning are addressed, achieving efficient and privacy-preserving model optimization and improving task accuracy and communication efficiency.
Patent Information
- Application Number
- CN202511191098.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-22
- Publication Date
- 2025-10-31
AI Technical Summary
In federated learning, the explosion of communication overhead in large models, the challenge of data heterogeneity, and the privacy-efficiency imbalance pose challenges to traditional methods in terms of communication efficiency, heterogeneity adaptability, and privacy protection.
A selective aggregation mechanism is adopted, which performs local training by receiving the pre-trained base model and the low-rank matrix of the local adapter. Only the first low-rank matrix responsible for general knowledge is transmitted, while the second low-rank matrix responsible for specific knowledge is retained locally. The weights are updated by combining the global low-rank matrix, thereby achieving model optimization.
It significantly improves communication efficiency, reduces communication load, enhances task accuracy under non-independent and identically distributed data, ensures privacy protection, and achieves zero-latency model expansion.
Smart Images

Figure CN120874970A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence, finance, and healthcare, and particularly to a method, apparatus, device, and medium for optimizing large models based on federated learning. Background Technology
[0002] With the deep application of artificial intelligence in data-sensitive fields such as healthcare and finance, federated learning has become a core technological paradigm due to its privacy-preserving characteristics. This technology allows multiple clients (such as hospitals and banks) to collaboratively optimize models through local training and parameter interaction without sharing raw data. However, the rise of large models (such as LLaMA and GPT series with billions of parameters) poses a serious challenge to traditional federated learning.
[0003] Explosive communication overhead: Large models have hundreds of millions to trillions of parameters, and frequent transmission of all model parameters between clients and servers leads to network bandwidth overload. For example, in a cross-hospital disease diagnosis system, a single round of communication can take several hours, which cannot meet real-time requirements.
[0004] Data heterogeneity challenge: Client data exhibits non-independent identically distributed (Non-IID) characteristics (e.g., hospital A contains only tumor data, hospital B contains only cardiovascular data), causing local models to deviate from the global optimum, and the accuracy of the aggregated model can decrease by more than 20%.
[0005] Privacy-efficiency imbalance: While communication compression (such as gradient sparsity) reduces bandwidth, it exacerbates convergence instability under heterogeneous data; personalized mechanisms (such as client adapters) require additional storage of local parameters, increasing deployment costs and failing to coordinate with pre-trained knowledge.
[0006] Therefore, there is a need to provide a large-scale model optimization scheme that can balance communication efficiency, heterogeneous adaptability, and privacy protection in a federated scenario. Summary of the Invention
[0007] This invention provides a method, apparatus, device, and medium for large model optimization based on federated learning, aiming to balance communication efficiency, heterogeneous adaptability, and privacy protection in large model optimization.
[0008] Firstly, a method for optimizing large models based on federated learning is provided, including the following steps:
[0009] S10. Receive a global model from the server, the global model including a pre-trained base model and a trainable local adapter, the local adapter including a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing local data-specific knowledge.
[0010] S20. Perform local training on the global model using local data, keeping the weights of the base model frozen during local training, and updating the first low-rank matrix and the second low-rank matrix;
[0011] S30. The updated first low-rank matrix is sent to the server so that the server aggregates the updated first low-rank matrix of each client to obtain a global low-rank matrix.
[0012] S40. Receive the global low-rank matrix from the server, generate weight update amounts to update the weights of the local adapter of the global model, and obtain the optimized global model.
[0013] Secondly, a large model optimization device based on federated learning is provided, including:
[0014] A global model acquisition module is used to receive a global model from a server. The global model includes a pre-trained base model and a trainable local adapter. The local adapter includes a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing local data-specific knowledge.
[0015] The local training module is used to perform local training on the global model using local data. During local training, the weights of the base model are frozen, and the first low-rank matrix and the second low-rank matrix are updated.
[0016] The matrix upload module is used to send the updated first low-rank matrix to the server, so that the server can aggregate the updated first low-rank matrix of each client to obtain the global low-rank matrix.
[0017] The weight update module receives the global low-rank matrix from the server, generates weight update values to update the weights of the local adapter of the global model, and obtains the optimized global model.
[0018] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described large model optimization method based on federated learning.
[0019] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described large model optimization method based on federated learning.
[0020] In the above-described scheme for optimizing large models based on federated learning, including the method, apparatus, computer device, and storage medium, a global model is received from a server. This global model includes a pre-trained base model and a trainable local adapter. The local adapter includes a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing knowledge specific to local data. The global model is then locally trained using local data, with the weights of the base model frozen during local training. The first and second low-rank matrices are updated. The updated first low-rank matrix is sent to the server, which aggregates the updated first low-rank matrices from each client to obtain a global low-rank matrix. Finally, the global low-rank matrix from the server is received, and weight update amounts are generated to update the weights of the local adapter of the global model, resulting in an optimized global model. The solution provided by this invention effectively improves communication efficiency through a selective aggregation mechanism. Only the first low-rank matrix responsible for general knowledge is transmitted, reducing the communication load of federated learning to a fraction of that of traditional methods and breaking through the bandwidth bottleneck of large model collaborative training. The strategy of preserving the client-specific second low-rank matrix locally effectively prevents the dilution effect of global aggregation on personalized knowledge and significantly improves the task accuracy under non-independent and identically distributed data. The functional decoupling architecture of the first and second low-rank matrices blocks the leakage path of local data distribution while ensuring basic semantic consistency through the global low-rank matrix, and the weight merging mechanism in the inference stage ensures zero-latency expansion. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a schematic diagram of an application environment for a large model optimization method based on federated learning in one embodiment of the present invention;
[0023] Figure 2 This is a schematic diagram of a large model optimization method based on federated learning in one embodiment of the present invention;
[0024] Figure 3 This is a schematic diagram of a weight update process in one embodiment of the present invention;
[0025] Figure 4 This is a schematic diagram of a large model optimization device based on federated learning in one embodiment of the present invention;
[0026] Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;
[0027] Figure 6 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0029] The large model optimization method based on federated learning provided in this invention is applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The client receives a global model from the server. This global model includes a pre-trained base model and a trainable local adapter. The local adapter includes a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing knowledge specific to local data. The client then performs local training on the global model using local data, keeping the weights of the base model frozen during local training, and updates the first and second low-rank matrices. The updated first low-rank matrix is sent to the server, allowing the server to aggregate the updated first low-rank matrices from each client to obtain a global low-rank matrix. Finally, the client receives the global low-rank matrix from the server, generates weight update amounts, and updates the weights of the local adapter of the global model to obtain an optimized global model. The solution provided in this invention effectively improves communication efficiency through a selective aggregation mechanism. Only the first low-rank matrix responsible for general knowledge is transmitted, reducing the communication load of federated learning to a negligible level compared to traditional methods and overcoming the bandwidth bottleneck of large-scale model collaborative training. The strategy of preserving the client-specific second low-rank matrix locally effectively prevents the dilution effect of global aggregation on personalized knowledge, significantly improving task accuracy under non-independent and identically distributed data. The functional decoupling architecture of the first and second low-rank matrices blocks local data distribution leakage paths while ensuring basic semantic consistency through the global low-rank matrix, and the weight merging mechanism during the inference phase ensures zero-latency expansion. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.
[0030] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a large model optimization method based on federated learning provided in an embodiment of the present invention includes the following steps:
[0031] S10. Receive a global model from the server. The global model includes a pre-trained base model and a trainable local adapter. The local adapter includes a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing local data-specific knowledge. The local data is local medical data or local financial data.
[0032] Step S10 further includes:
[0033] The first low-rank matrix is initialized with a random Gaussian distribution, and the second low-rank matrix is initialized with a zero matrix.
[0034] This embodiment uses the training of a cross-hospital stroke prediction model in the medical field as an example. A tertiary hospital (client A) has a dataset of 100,000 brain MRI images, a specialized hospital (client B) stores 80,000 cerebral angiography reports, and a community hospital (client C) has accumulated 120,000 long-term rehabilitation records of patients. The data distribution of each institution exhibits a significant non-independent identically distributed (Non-IID) characteristic: client A's image data focuses on acute-phase lesion features, client B contains a large number of vascular malformation annotations, and client C is rich in drug response time-series data. If traditional federated learning directly aggregates all model parameters (such as RoBERTa-base with 390 million parameters), a single round of communication requires the transmission of 1.2GB of data, and due to data heterogeneity, the accuracy of the aggregated model on the sensitivity detection task of client A decreases by 21%.
[0035] The server selects the pre-trained clinical language model ClinicalBERT as the base model (110 million parameters) and freezes all its weights. Low-rank matrix pairs are initialized for each hospital:
[0036] The first low-rank matrix (A matrix): dimension 256×512, initialized with a Gaussian distribution with a standard deviation of 0.02, is responsible for learning common medical knowledge across hospitals (such as semantic associations of common stroke symptoms such as "limb numbness" and "speech disorder").
[0037] The second low-rank matrix (B matrix): dimension 1024×256, initialized as an all-zero matrix, is used to capture the hospital's unique knowledge (such as the "diffusion-weighted imaging high signal" feature of client A, and the "vascular stenosis grading" quantification rules of client B).
[0038] After initialization, the server distributes the basic model weights and matrix initialization rules to each hospital. When storing the B matrix, the local system of client B enables the hardware encryption module to ensure that patient privacy data cannot be reverse-analyzed through the matrix.
[0039] S20. The global model is locally trained using local data. During local training, the weights of the base model are frozen, and the first low-rank matrix and the second low-rank matrix are updated.
[0040] Specifically, in step S20, updating the first low-rank matrix and the second low-rank matrix includes:
[0041] The first low-rank matrix is updated to learn cross-client common features, and the second low-rank matrix is updated to learn local data specific features.
[0042] In this embodiment, client A uses a local MRI report dataset for training:
[0043] Fixed base model weights: ClinicalBERT's 110 million parameters are treated as read-only data, and are only called without being updated during forward propagation.
[0044] Dual-matrix collaborative update:
[0045] Matrix A learns common features through backpropagation: for example, it establishes cross-institutional association rules between "blood pressure > 180 mmHg" and "hemorrhagic stroke";
[0046] Matrix B focuses on optimizing the hospital's unique characteristics: it strengthens the weighting of features specific to MRI images, such as "disappearance of the insula sign" and "shallow sulci".
[0047] Dynamic weight generation: The low-rank adapter output $ΔW = B × A$ is calculated in real time during training, allowing the prediction results to simultaneously reflect general medical knowledge and the hospital's experience in imaging diagnosis. This scheme uses only 18% of the memory required for training compared to full parameter fine-tuning, and client A's AUC is improved to 0.91 on the local validation set (compared to only 0.79 for traditional federated learning schemes).
[0048] S30. The updated first low-rank matrix is sent to the server so that the server aggregates the updated first low-rank matrix of each client to obtain a global low-rank matrix.
[0049] Specifically, step S30 includes:
[0050] The updated first low-rank matrix is sent to the server so that the server can perform a weighted average aggregation of the updated first low-rank matrices from each client to obtain a global low-rank matrix.
[0051] Step S30 further includes:
[0052] Store the updated second low-rank matrix locally.
[0053] In this embodiment, client A only uploads the updated A matrix (file size 2.4MB) to the server, while the B matrix, containing the hospital's imaging diagnostic key, is stored in the isolated storage area of the local medical data platform. Compared to the traditional solution that requires uploading 430MB of model parameters, the communication overhead is reduced to 0.56%. The server adopts a weighted average aggregation strategy: weights are allocated according to the amount of data from each hospital (client A: 30%, B: 25%, C: 45%) to generate a global A matrix. This matrix has incorporated shared knowledge among the three institutions, such as identifying "fasting blood glucose > 7.0 mmol / L" as a cross-institutional risk factor.
[0054] S40. Receive the global low-rank matrix from the server, generate weight update amounts to update the weights of the local adapter of the global model, and obtain the optimized global model.
[0055] Among them, such as Figure 3 As shown, step S40 specifically includes:
[0056] S41, Receive the global low-rank matrix sent by the server;
[0057] S42. Perform matrix multiplication on the global low-rank matrix and the updated second low-rank matrix to obtain the generated weight update amount;
[0058] S43. Update the weights of the local adapter with the updated weights;
[0059] S44. Merge the updated weights of the local adapter with the frozen base model weights to obtain the optimized global model.
[0060] In this embodiment, client A performs a key operation after receiving the global matrix A:
[0061] Matrix replacement: Overwrite the local original A matrix with the global A matrix. This operation enables the hospital model to immediately acquire cross-hospital common knowledge.
[0062] Adapter Reconstruction: Multiply the global A matrix by the local B matrix to generate a new adapter weight $ΔW_{new}=B_A\times A_{global}$.
[0063] Zero-latency merging: $ΔW_{new}$ is superimposed with the frozen ClinicalBERT weights to form the final inference model. The merging process is completed at the GPU driver layer, and the inference latency is kept within 7ms (consistent with the original model).
[0064] The refactored model performs as follows in client A's tests:
[0065] Improved general metrics: The accuracy rate for identifying typical symptoms reached 95.2% (compared to 89.7% before aggregation);
[0066] Personalized capabilities retained: The hospital's unique imaging feature detection accuracy remains at a high level of 98.3%;
[0067] Privacy and security: Differential privacy testing shows that the global A matrix cannot infer the original data distribution of any hospital. This invention discloses a large model optimization method based on federated learning, which can also be applied to fields such as banking, finance, and insurance.
[0068] For example, in a joint risk control scenario involving banks and insurance companies:
[0069] A financial institution comprises a commercial banking division (client A), a life insurance subsidiary (client B), and a wealth management platform (client C). Regulatory requirements necessitate collaborative prevention and control of cross-business financial risks under conditions of data segregation.
[0070] The server deploys a pre-trained FinBERT financial model (870 million parameters) as the base model, employing a hardware-level write-protection mechanism to freeze its weights. When initializing low-rank matrix pairs:
[0071] The first low-rank matrix (A matrix) is set to 384×768 and initialized with a Gaussian distribution (standard deviation 0.015). It is used to capture common risk patterns across sectors, such as financial crime characteristics like "short-term multi-account closed-loop fund flow" or "abnormal time window trading clusters".
[0072] The second low-rank matrix (B matrix) has a dimension of 1024×384. It is initialized with zero values and bound to a hardware security module, with differentiated configurations for each client: the commercial bank client focuses on modeling the "small-amount test-large-amount transfer" behavior sequence of credit cards; the life insurance client constructs the probability distribution characteristics of the "insurance-claim time interval"; and the wealth management platform client extracts the association rules of "virtual currency-cross-border accounts". The initialization information is encrypted and transmitted using the national cryptographic algorithm SM4, and the matrix seed value is generated locally by each client to ensure that the original data does not leave the domain.
[0073] Each client updates the low-rank matrix based on its private data:
[0074] Commercial bank clients encode transaction logs into a time-series feature matrix and extract local patterns such as "a sudden increase in the frequency of interbank transfers within 24 hours" through a one-dimensional convolutional layer. During backpropagation, matrix A learns common rules across institutions (such as the standard deviation threshold of transaction amount), while matrix B optimizes the credit card cash-out detection model and uses adversarial training to enhance robustness against new criminal methods.
[0075] The life insurance client integrates structured policy data with unstructured medical texts. The FinBERT basic model extracts key entities from medical reports (such as contradictions in injury diagnosis). The B matrix establishes a causal relationship between "frequency of medical visits and beneficiary liabilities" through a latent variable model. Gaussian noise is added during the training process to meet differential privacy requirements.
[0076] The wealth management platform client integrates a graph neural network to construct a heterogeneous graph of customer transactions. The rank parameter of the B matrix is dynamically adjusted to adapt to the needs of suspicious transaction detection, and graphics card computation graph solidification technology is used to accelerate training. After local optimization, the credit card fraud detection rate of commercial banks has increased to 96.5%, while the memory usage is only 2% of that of full parameter fine-tuning.
[0077] Parameter optimization before client upload:
[0078] Value screening: Calculate the Fisher information of neurons in matrix A and retain only the subset of parameters with the highest importance;
[0079] Layered compression: Low-precision quantization from FP32 to FP16 is used to reduce the data volume by 50%, combined with 8×8 block structured pruning to further compress by 40%, and finally, a homomorphic encryption library is used to protect transmission security.
[0080] Server aggregation: Based on logarithmic weighting of sample size (commercial banks 0.45, life insurance 0.30, wealth management platforms 0.25), a global A matrix integrating cross-business knowledge is generated, for example, to identify a criminal chain of "credit card cash-out → high-value insurance → rapid claims settlement". This process reduces the single-round communication volume from 3.1GB to 14.7MB, improving transmission efficiency by 250 times.
[0081] After the client receives the global A matrix:
[0082] Hot-swap mechanism: Through double buffering of video memory and asynchronous transmission technology, the atomic replacement of the global A matrix is completed within 0.5 milliseconds;
[0083] Real-time adapter reconstruction: Multiply the local B matrix with the global A matrix to generate new weight increments, and accelerate the computation to 312 TFLOPS using tensor cores;
[0084] Incremental risk decision-making: When a transaction request is input, the base model outputs a baseline risk assessment, and the local adapter injects real-time feature increments. For example, if a user's behavior chain across multiple clients is detected: a commercial bank triggers credit card cash-out rules (B-matrix proprietary features), a life insurance company marks short-term high-coverage insurance purchases (global A-matrix knowledge), and a wealth management platform associates virtual currency money laundering (global A-matrix knowledge), the system generates a comprehensive risk score and intercepts the transaction within 12 milliseconds.
[0085] This solution achieves three core breakthroughs:
[0086] Improved risk control accuracy: The detection rate of cross-business financial crimes reached 94.3% (28.6 percentage points higher than traditional federated learning), and the false alarm rate dropped to 1.9%.
[0087] Efficiency optimization: Communication overhead is reduced by 99.5%, model update latency is shortened from 6 hours to 90 seconds, and energy consumption for single transaction analysis is reduced by 96%;
[0088] Compliance Assurance: The matrix separation mechanism meets the regulatory requirement of "data not leaving the domain," ensuring that original sensitive information is retained in a local secure zone throughout the entire process. This solution has passed the Level 3 Financial Data Security Certification, providing key technical support for collaborative risk control in banking and insurance.
[0089] In summary, the embodiments of the present invention effectively improve communication efficiency through a selective aggregation mechanism, transmitting only the first low-rank matrix responsible for general knowledge, reducing the communication load of federated learning to a negligible level compared to traditional methods, and breaking through the bandwidth bottleneck of large model collaborative training; the strategy of preserving the client-specific second low-rank matrix locally effectively prevents the dilution effect of global aggregation on personalized knowledge, significantly improving task accuracy under non-independent and identically distributed data; the functional decoupling architecture of the first and second low-rank matrices blocks the leakage path of local data distribution while ensuring basic semantic consistency through the global low-rank matrix, and the weight merging mechanism in the inference stage ensures zero-latency expansion.
[0090] like Figure 4 As shown, this embodiment of the invention provides a large model optimization device based on federated learning, comprising:
[0091] The global model acquisition module 10 is used to receive a global model from the server. The global model includes a pre-trained base model and a trainable local adapter. The local adapter includes a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing local data-specific knowledge.
[0092] The global model acquisition module 10 is also used to initialize the first low-rank matrix with a random Gaussian distribution and to initialize the second low-rank matrix with a zero matrix.
[0093] The local training module 20 is used to perform local training on the global model using local data. During local training, the weights of the base model are frozen, and the first low-rank matrix and the second low-rank matrix are updated.
[0094] The local training module 20 is also used to update the first low-rank matrix to learn cross-client general features and update the second low-rank matrix to learn local data-specific features.
[0095] The matrix upload module 30 is used to send the updated first low-rank matrix to the server so that the server can aggregate the updated first low-rank matrix of each client to obtain a global low-rank matrix.
[0096] The matrix upload module 30 is specifically used to send the updated first low-rank matrix to the server, so that the server can perform weighted average aggregation of the updated first low-rank matrix of each client to obtain a global low-rank matrix.
[0097] The matrix upload module 30 is also used to store the updated second low-rank matrix locally.
[0098] The weight update module 40 is used to receive the global low-rank matrix from the server, generate weight update amounts to update the weights of the local adapter of the global model, and obtain the optimized global model.
[0099] Weight update module 40 is specifically used for:
[0100] Receive the global low-rank matrix sent by the server;
[0101] The updated weights are generated by performing matrix multiplication on the global low-rank matrix and the updated second low-rank matrix.
[0102] Update the weights of the local adapter with the updated weights;
[0103] The updated weights of the local adapter are merged with the frozen base model weights to obtain the optimized global model.
[0104] Specific limitations regarding the large model optimization apparatus based on federated learning can be found in the limitations of the large model optimization method based on federated learning mentioned above, and will not be repeated here. Each module in the aforementioned large model optimization apparatus based on federated learning can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0105] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements server-side functions or steps of a large model optimization method based on federated learning.
[0106] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a large model optimization method based on federated learning.
[0107] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:
[0108] S10. Receive a global model from the server, the global model including a pre-trained base model and a trainable local adapter, the local adapter including a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing local data-specific knowledge.
[0109] S20. Perform local training on the global model using local data, keeping the weights of the base model frozen during local training, and updating the first low-rank matrix and the second low-rank matrix;
[0110] S30. The updated first low-rank matrix is sent to the server so that the server aggregates the updated first low-rank matrix of each client to obtain a global low-rank matrix.
[0111] S40. Receive the global low-rank matrix from the server, generate weight update amounts to update the weights of the local adapter of the global model, and obtain the optimized global model.
[0112] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0113] S10. Receive a global model from the server, the global model including a pre-trained base model and a trainable local adapter, the local adapter including a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing local data-specific knowledge.
[0114] S20. Perform local training on the global model using local data, keeping the weights of the base model frozen during local training, and updating the first low-rank matrix and the second low-rank matrix;
[0115] S30. The updated first low-rank matrix is sent to the server so that the server aggregates the updated first low-rank matrix of each client to obtain a global low-rank matrix.
[0116] S40. Receive the global low-rank matrix from the server, generate weight update amounts to update the weights of the local adapter of the global model, and obtain the optimized global model.
[0117] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0120] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for optimizing large models based on federated learning, characterized in that, Includes the following steps: S10. Receive a global model from the server, the global model including a pre-trained base model and a trainable local adapter, the local adapter including a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing local data-specific knowledge. S20. Perform local training on the global model using local data, keeping the weights of the base model frozen during local training, and updating the first low-rank matrix and the second low-rank matrix; S30. The updated first low-rank matrix is sent to the server so that the server aggregates the updated first low-rank matrix of each client to obtain a global low-rank matrix. S40. Receive the global low-rank matrix from the server, generate weight update amounts to update the weights of the local adapter of the global model, and obtain the optimized global model.
2. The large model optimization method based on federated learning as described in claim 1, characterized in that, Step S10 also includes: The first low-rank matrix is initialized with a random Gaussian distribution, and the second low-rank matrix is initialized with a zero matrix.
3. The large model optimization method based on federated learning as described in claim 1, characterized in that, In step S20, updating the first low-rank matrix and the second low-rank matrix specifically includes: The first low-rank matrix is updated to learn cross-client common features, and the second low-rank matrix is updated to learn local data specific features.
4. The large model optimization method based on federated learning as described in claim 1, characterized in that, Step S30 specifically includes: The updated first low-rank matrix is sent to the server so that the server can perform a weighted average aggregation of the updated first low-rank matrices from each client to obtain a global low-rank matrix.
5. The large model optimization method based on federated learning as described in claim 4, characterized in that, Step S30 also includes: Store the updated second low-rank matrix locally.
6. The large model optimization method based on federated learning as described in claim 1, characterized in that, Step S40 specifically includes: Receive the global low-rank matrix sent by the server; The updated weights are generated by performing matrix multiplication on the global low-rank matrix and the updated second low-rank matrix. Update the weights of the local adapter with the updated weights; The updated weights of the local adapter are merged with the frozen base model weights to obtain the optimized global model.
7. The large model optimization method based on federated learning as described in any one of claims 1-6, characterized in that, The local data refers to local medical data or local financial data.
8. A large model optimization device based on federated learning, characterized in that, include: A global model acquisition module is used to receive a global model from a server. The global model includes a pre-trained base model and a trainable local adapter. The local adapter includes a first low-rank matrix for learning general knowledge and a second low-rank matrix for capturing local data-specific knowledge. The local training module is used to perform local training on the global model using local data. During local training, the weights of the base model are frozen, and the first low-rank matrix and the second low-rank matrix are updated. The matrix upload module is used to send the updated first low-rank matrix to the server, so that the server can aggregate the updated first low-rank matrix of each client to obtain the global low-rank matrix. The weight update module receives the global low-rank matrix from the server, generates weight update values to update the weights of the local adapter of the global model, and obtains the optimized global model.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the large model optimization method based on federated learning as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the large model optimization method based on federated learning as described in any one of claims 1 to 7.
Citation Information
Cited By
Federal remote sensing large model training method and system based on low-rank self-adaption
CN121438126A
A federated remote sensing large model training method and system based on low-rank adaptation
CN121438126B
