A federated low-rank fine-tuning method and system applied to a medical data collaborative modeling scene

Abstract

This invention discloses a federated low-rank fine-tuning method and system for collaborative modeling of medical data. The system includes a server node and client devices distributed across multiple medical institutions. Each medical institution fine-tunes a pre-trained medical model locally without sharing the original medical data, and uploads the generated low-rank model parameter increments to the server node. The server node aggregates the parameter increments from multiple medical institutions and performs low-rank reconstruction on the aggregation results based on randomized singular value decomposition. Simultaneously, the reconstructed residuals are cached and merged and updated in subsequent training rounds. Through this approach, multiple medical institutions can jointly train the model while protecting patient privacy, achieve structured processing of model parameter increments, reduce communication and computational overhead, and improve the stability and practicality of model training.

Description

Technical Field

[0001] This invention relates to the fields of distributed machine learning and medical information systems, specifically to a federated low-rank fine-tuning method and system for jointly training models across multiple medical institutions. It aims to solve the problems of structural destruction, excessive aggregation noise, and failure to retain training momentum in existing federated low-rank fine-tuning methods, thereby significantly improving the stability and performance of large models for cross-device collaborative fine-tuning. Background Technology

[0002] In healthcare information systems, large amounts of clinical data, imaging data, and electronic medical records are stored scattered across different hospitals or medical institutions. Due to patient privacy protection, data security regulations, and restrictions on cross-institutional data sharing, medical data often cannot be centrally aggregated for unified model training.

[0003] To improve the generalization ability of medical models while protecting patient privacy, the industry has proposed using federated learning, where multiple medical institutions train models independently on local data and only upload model parameters or parameter increments to a central server for aggregation. However, in actual medical system deployment, existing federated learning methods still have the following technical problems: First, medical models typically have a large number of parameters, and uploading these parameters can put significant pressure on the network bandwidth between the hospital and the central server, affecting system operating efficiency. Secondly, different hospitals have different patient groups, disease distributions, and data collection methods. Simply averaging the model parameters uploaded by each hospital can easily lead to unstable model updates and affect the reliability of the model in real medical scenarios. Third, when updating medical models using Low-Rank Adaptation (LoRA) fine-tuning, existing federated aggregation methods struggle to maintain the overall structural characteristics of parameter increments, further amplifying aggregation bias caused by cross-hospital data differences. Specifically, when applying LoRA to a federated environment, existing technologies suffer from the following key drawbacks: 1. Biased aggregation introduces noise.

[0004] LoRA represents parameter changes as in , and The aim is to reduce the amount of parameter training by utilizing the low-rank representation of the parameter matrix. If the server-side still uses the FedAvg aggregation method to aggregate matrices B and A separately, that is: However, matrix multiplication does not possess the property of linear additivity, that is: This results in the aggregated matrix being mixed with a large amount of random noise, which destroys the original LoRA's expressive power.

[0005] Studies have shown that such structural disruptions are particularly severe in tasks such as mathematical reasoning and code generation, leading to numerical instability and performance degradation.

[0006] 2. Unbiased polymerization leads to the loss of the LoRA structure.

[0007] Although some studies have attempted to introduce unbiased aggregation strategies (e.g., through more complex weighting or modification), unbiased aggregation still cannot guarantee the integrity of the LoRA structure. Even if the parameter updates after aggregation are numerically unbiased, the loss of low-rank structure still exists. Since the multiplication order of matrix factorization cannot guarantee the preservation of low-rank structure, this method cannot effectively recover the original low-rank representation of LoRA, resulting in the inability to propagate the low-rank structure during model training and a significant performance degradation.

[0008] 3. Current research lacks a unified solution for preserving low-rank structures.

[0009] While some methods attempt to mitigate these issues by adjusting aggregation strategies, optimizing algorithms, or enhancing local updates, existing techniques still cannot simultaneously maintain the low-rank structure of LoRA and transfer optimizer momentum, thus failing to address the problems of aggregation noise and structure loss. Therefore, a new technical solution is urgently needed that can effectively maintain the low-rank structure, reduce aggregation noise, and preserve training momentum to improve the training efficiency and stability of large-scale models in federated learning.

[0010] Therefore, there is an urgent need for a federated fine-tuning scheme that can simultaneously ensure the integrity of low-rank structures, suppress aggregation noise, and achieve implicit momentum preservation, so as to improve the practicality, stability, and performance of medical consortium modeling. Summary of the Invention

[0011] To address the common problems in fine-tuning of low-rank adaptation (LoRA)-based medical models in existing medical information systems, such as the introduction of aggregation noise, loss of low-rank structure, and inability to maintain training momentum across epochs, which easily lead to training instability, slow convergence, and even performance degradation, this invention proposes a momentum-preserving parameter aggregation method for federated low-rank fine-tuning. This method performs structured decomposition and reconstruction of low-rank parameter increments uploaded from multiple clients on the server side. While strictly maintaining the low-rank constraints of LoRA, it achieves implicit momentum preservation by accumulating residuals across epochs. Therefore, without increasing the communication and computational burden on clients, it significantly improves the stability and convergence efficiency of federated low-rank fine-tuning.

[0012] The technical solution of the present invention is as follows: This invention constructs a collaborative update process between clients and servers within a federated learning framework. On the client side, each client only performs low-rank fine-tuning on the LoRA parameters in the model, while keeping the base model parameters frozen. After each training round, only the corresponding low-rank increment matrix is ​​uploaded, effectively reducing communication overhead and protecting local data privacy. On the server side, for low-rank increment matrices from multiple clients, this invention does not employ a direct weighted average aggregation method. Instead, it first performs singular value decomposition on the aggregated global increment matrix, extracting its principal spectral components, and then truncating the principal singular values ​​and... The equilibrium strategy reconstructs the global LoRA parameters that satisfy the preset rank constraint, mathematically ensuring that the aggregation result strictly falls within the low-rank subspace, thus avoiding the structural noise and degradation of expressive power introduced by traditional aggregation methods.

[0013] Building upon this foundation, this invention further analyzes the energy proportion of the spectral information obtained from singular value decomposition (SVD), extracting out-of-spectral components not represented by the low-rank reconstruction as residual terms, which are then accumulated and propagated between adjacent rounds of federated training. By fusing these residual terms with the base model parameters, this invention achieves a continuous and stable optimization direction for global model updates across multiple rounds of federated training. Simultaneously, to improve server-side computational efficiency in large-scale models and multi-client scenarios, this invention introduces a customized randomized SVD method, significantly reducing the computational complexity of decomposition while ensuring approximately lossless low-rank recovery, thereby enhancing the method's engineering scalability.

[0014] The technical solution of the present invention is as follows: A federated low-rank fine-tuning method for collaborative modeling of medical data, deployed in a federated learning system comprising one server node and multiple client devices, is characterized by comprising the following steps: In each round of federal training At the beginning, the server sends the current global LoRA parameters to each client. and residuals accumulated from historical rounds ; The residuals received by each client These parameters are incorporated into the local base model parameters to obtain enhanced local base parameters. Each client, under the condition of freezing the enhanced base model parameters, uses local private data to adjust the global LoRA parameters. Perform low-rank fine-tuning to obtain the updated local low-rank parameters; Each client will update the local low-rank increment corresponding to the local update parameter. Uploaded to the server; After receiving the low-rank increments from all clients, the server aggregates them to obtain the global increment matrix. The formula is as follows: ; In the formula, This indicates the number of clients participating in the current federated training round; Indicates the client index; Indicates the first The downprojection matrix obtained by local low-rank fine-tuning on each client; Indicates the first The upprojection matrix obtained by local low-rank fine-tuning on each client; Indicates the first The low-rank parameter increment matrix corresponding to each client; This represents the global parameter increment matrix obtained by aggregating the low-rank parameter increments of all clients on the server side. The server performs singular value decomposition on the global incremental matrix to obtain singular values. The formula is as follows: In the formula, Describes a left singular vector matrix, whose column vectors Global increment matrix The One left singular vector; Describes a right singular vector matrix, whose column vectors Global increment matrix The One right singular vector; This represents a singular value diagonal matrix, whose diagonal elements For the corresponding number There are singular values, and satisfy ; This represents the number of singular values ​​obtained after singular value decomposition. Indicates by the first A rank-one matrix consisting of singular vector pairs; The global increment matrix is ​​represented in the th order. The spectral components in each singular direction. The singular value decomposition represents the global increment matrix as the sum of several rank-one matrices ordered by energy, which are used for subsequent low-rank reconstruction and residual extraction; The singular values ​​are sorted, the first r principal singular values ​​are truncated, and the new global LoRA parameters for the next round are reconstructed, as shown in the following formula: In the formula, Indicates the index of the current federated training round; This represents the preset low-rank reconstruction rank value; The first part represents the result obtained from the singular value decomposition of the global increment matrix. A matrix consisting of left singular vectors; The first part represents the result obtained from the singular value decomposition of the global increment matrix. A matrix consisting of right singular vectors; Indicates from the previous A diagonal matrix composed of principal singular values; This represents the diagonal matrix obtained by taking the square root of each element of the principal singular value in the diagonal matrix; Indicates the first The global LoRA projection parameter matrix obtained by reconstruction during rounds of federated training; Indicates the first The global LoRA projection parameter matrix obtained by reconstruction during rounds of federated training; The server analyzes the energy proportion of singular values ​​and extracts the (r+1)th to (r+s)th singular values ​​and their corresponding singular vectors to construct the residual term for the current round. In the formula, This indicates the number of spectral components selected for constructing the residual term; This represents the residual matrix constructed in the current federated training round, used to characterize the residual terms that were not previously... The extraspectral components covered by each main spectral component; This represents the result of the singular value decomposition of the global increment matrix. to A matrix consisting of left singular vectors; This represents the first singular value decomposition result of the global increment matrix. to A matrix consisting of right singular vectors; Indicates by the first to A diagonal matrix consisting of singular values; Representation matrix The transpose matrix of . Wherein, the first to The spectral components corresponding to each singular value and its corresponding singular vector reflect the energy in the global increment matrix that was not retained by the low-rank reconstruction. This energy is used for subsequent cross-round accumulation to achieve continuity and stability in the training process. The updated residuals accumulated across rounds and the new global LoRA parameters are sent to the client for use in the next round of federated training.

[0015] Furthermore, the singular value decomposition specifically includes: Generate random Gaussian matrices Its dimensions are ,in Represents the global parameter increment matrix The column dimension is used to characterize the original feature dimension of the parameter space; represents the dimension of the random projected subspace, used to control the size of the subspace retained when approximating singular value decomposition, where Not less than the preset low-rank reconstruction rank value; Calculate intermediate variables ; For intermediate variables Perform QR matrix decomposition to obtain an orthogonal basis. ,Right now ; global increment matrix Map to a low-dimensional subspace constructed by Q and then perform SVD decomposition: Through calculation get The left singular value matrix is ​​obtained by decomposing the matrix into a complete matrix factorization. .

[0016] Furthermore, the formula for the singular value energy proportion analysis is as follows: In the formula, Indicates the preceding The cumulative energy percentage of each singular value corresponding to a spectral component; This represents the number of singular values ​​involved in the energy accumulation calculation, where ; This represents the first element of the global parameter increment matrix after singular value decomposition, arranged in descending order. One singular value; This represents the first element of the global parameter increment matrix after singular value decomposition, arranged in descending order. One singular value; Indicates the preset rank value used for low-rank reconstruction; This indicates the number of clients participating in the current federated training round; This represents the theoretical maximum effective upper bound of the global parameter increment matrix in the current federated training round.

[0017] Furthermore, the client is a terminal device or a medical institution server, and the server is a central server or a cloud computing node.

[0018] A federated low-rank fine-tuning system for implementing the above method is characterized by comprising: Multiple client devices, each client device including: The module includes a parameter receiving module, a residual fusion module, a low-rank fine-tuning module, a parameter increment generation module, and a parameter uploading module. Server nodes include: The system includes a parameter receiving module, a parameter aggregation module, a randomized singular value decomposition module, a low-rank reconstruction module, a residual extraction and caching module, and a parameter distribution module. Each client device is connected to the server node via a communication network, and the server node is configured to execute the above method.

[0019] The present invention also provides an electronic device, characterized in that it includes: One or more processors; Memory, which stores one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 4.

[0020] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method described above.

[0021] The technical effects of this invention are as follows: 1) To address the common problem of low-rank structure collapse caused by biased aggregation noise and unbiased aggregation in federated LoRA fine-tuning, this invention fundamentally restores and maintains the low-rank expressive power of LoRA through a structured aggregation and low-rank reconstruction mechanism based on singular value decomposition, avoids the destruction of directional gradient information during aggregation, and keeps the federated training process stable in multiple iterations.

[0022] 2) By introducing a residual accumulation mechanism based on spectral energy analysis, this invention eliminates the noise problem in the LoRA aggregation process without changing the local training process, realizes implicit momentum preservation across rounds, effectively alleviates the common convergence oscillation and training discontinuity problems in federated optimization, and significantly improves the convergence speed and final performance of the model in heterogeneous data scenarios.

[0023] 3) By combining randomized singular value decomposition with low-rank aggregation, this invention significantly reduces the computation and time overhead on the server side while ensuring the accuracy of low-rank structure recovery. This makes the method applicable to large-scale language models and federated fine-tuning scenarios involving multiple clients, and has good engineering scalability and practical deployment value.

[0024] In summary, this invention provides a parameter aggregation scheme for federated low-rank fine-tuning that balances theoretical rigor with engineering feasibility, and has significant advantages in aggregation stability, training efficiency, and model performance. Attached Figure Description

[0025] Figure 1This is a flowchart of the momentum-preserving parameter aggregation method for federated low-rank fine-tuning in a medical information system.

[0026] Figure 2 This is a modular framework diagram of the momentum-preserving parameter aggregation method for federated low-rank fine-tuning in medical information systems. Detailed Implementation

[0027] To better understand the embodiments of the present invention, further descriptions are provided below with reference to specific implementation examples. It is worth noting that the following examples are for illustrative purposes only and do not limit the implementation of the present invention.

[0028] Referring to the overall process shown in the attached diagram, in a specific embodiment, the federated learning system includes a central server node and client devices deployed within multiple hospitals. Each hospital, without sharing raw patient data, locally fine-tunes its pre-trained medical model and uploads the generated low-rank parameter increments to the central server. The central server aggregates and reconstructs the low-rank parameters from multiple hospitals and distributes the updated parameters to the client devices of each hospital, thereby enabling joint training of the medical model by multiple hospitals.

[0029] Each hospital's client-side devices are deployed within the hospital's internal medical information system to store and process medical data generated within that hospital, including but not limited to medical imaging data, electronic medical record data, or clinical examination data. This medical data is always stored within each hospital and is not transmitted across hospitals or institutions.

[0030] During model training, the central server node first distributes the pre-trained medical model and corresponding low-rank fine-tuning parameters to each hospital's client devices. Each hospital's client devices then perform local fine-tuning training on the medical model based on medical data stored within their hospital, generating incremental model parameters for the corresponding network layers. These incremental parameters are represented as low-rank matrices and uploaded to the central server node via the communication network.

[0031] After receiving low-rank parameter increments from multiple hospitals, the central server node performs weighted aggregation on these increments to obtain a global parameter increment matrix. Subsequently, the central server node performs randomized singular value decomposition on the global parameter increment matrix and reconstructs the decomposition results into a low-rank matrix according to preset rank constraints to obtain a low-rank global parameter update matrix.

[0032] Furthermore, the central server node calculates the residual matrix between the global parameter increment matrix and the low-rank global parameter update matrix, and caches the residual matrix in the server node. In subsequent training rounds, the central server node merges and updates the residual matrix with the parameter increment obtained from the new round of aggregation, and then distributes the updated low-rank parameters to the client devices of each hospital.

[0033] The above method enables collaborative training of medical models among multiple hospitals without sharing original medical data, reducing cross-hospital communication overhead and improving the stability of the model training process. It is suitable for medical application scenarios with high requirements for patient privacy protection.

[0034] The following details each step: Before federated training begins, the server initializes the global LoRA parameters. And these parameters along with the initial residual term =0 is sent to all participating client devices.

[0035] S1, Client-side low-rank fine-tuning: Each client Receive global LoRA parameters sent by the server and residual terms The LoRA parameters are replaced with the corresponding local parameters, and the residual terms are merged into the local base model parameters. With the base model parameters frozen, the client performs low-rank fine-tuning of the LoRA parameters based on local medical data to obtain locally updated parameters. ; S2, Parameter Upload: Each Client The low-rank increment matrix corresponding to the local update parameters Uploaded to the server; S3, Server-side Aggregation: The server receives low-rank increment matrices from multiple clients and aggregates them to obtain the global increment matrix. S4. Server-side singular value decomposition and low-rank reconstruction: The server performs singular value decomposition on the global incremental matrix to obtain... The singular values ​​are then sorted, and the first r principal singular values ​​are extracted. The global LoRA parameters are then reconstructed as follows: To accelerate the singular value decomposition calculation, the decomposition process includes the following sub-processes: S41. First, generate a random Gaussian matrix. And calculate intermediate variables. , where is the dimension to be preserved to achieve the approximate target. Due to the parameter structure we aggregate... in The rank of the matrix is ​​r, so The rank of the algorithm is no more than nr. Therefore, setting c=nr allows for lossless approximation calculations with higher computational efficiency.

[0036] S42. Perform QR matrix decomposition on Y to obtain an orthogonal basis. ,Right now Then Map to a low-dimensional subspace constructed by Q and then perform SVD decomposition: S43. Finally, it can be approximated by calculation. get The left singular value matrix is ​​obtained, thus yielding the complete matrix decomposition. .

[0037] S5. Residual Extraction and Momentum Preservation: Based on singular value energy proportion analysis, the server extracts the (r+1)th to (r+s)th singular values ​​and their corresponding singular vectors, and constructs residual terms. The residual terms and the reconstructed LoRA parameters are then sent to the client for the next round of federated training. This maintains the low-rank structure of LoRA while preserving momentum during the federated training process through residual accumulation across rounds.

[0038] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. Where there is no conflict, the above embodiments and features described therein can be combined with each other.

Claims

1. A federated low-rank fine-tuning method for collaborative modeling of medical data, deployed in a federated learning system comprising one server node and multiple client devices, characterized in that, The method includes the following steps: In each round of federal training At the beginning, the server sends the current global LoRA parameters to each client. and residuals accumulated from historical rounds ; The residuals received by each client These parameters are incorporated into the local base model parameters to obtain enhanced local base parameters. Each client, under the condition of freezing the enhanced base model parameters, uses local private data to adjust the global LoRA parameters. Perform low-rank fine-tuning to obtain the updated local low-rank parameters; Each client will update the local low-rank increment corresponding to the local update parameter. Uploaded to the server; After receiving the low-rank increments from all clients, the server aggregates them to obtain the global parameter increment matrix. The formula is as follows: ； In the formula, This indicates the number of clients participating in the current federated training round; Indicates the client index; Indicates the first The low-rank parameter increment matrix corresponding to each client; Indicates the first The downprojection parameter matrix obtained by local low-rank fine-tuning on each client; Indicates the first The upprojection parameter matrix obtained by local low-rank fine-tuning on each client; The server performs singular value decomposition on the global incremental matrix to obtain singular values. The formula is as follows: In the formula, Describes a left singular vector matrix, whose column vectors Global parameter increment matrix The One left singular vector; Describes a right singular vector matrix, whose column vectors Global parameter increment matrix The One right singular vector; This represents a singular value diagonal matrix, whose diagonal elements For the corresponding number There are singular values, and satisfy ; This represents the number of singular values ​​obtained after singular value decomposition. Indicates by the first A rank-one matrix consisting of singular vector pairs; The global parameter increment matrix is ​​represented in the th order. Spectral components in a singular direction; The singular values ​​are sorted, the first r principal singular values ​​are truncated, and reconstructed for the next round, i.e., the rth singular value. The global LoRA projection parameter matrix obtained from the reconstruction during the federated training round is given by the following formula: In the formula, Indicates the index of the current federated training round; This represents the preset low-rank reconstruction rank value; The first part represents the result obtained from the singular value decomposition of the global increment matrix. A matrix consisting of left singular vectors; The first part represents the result obtained from the singular value decomposition of the global increment matrix. A matrix consisting of right singular vectors; Indicates from the previous A diagonal matrix composed of principal singular values; This represents the diagonal matrix obtained by taking the square root of each element of the principal singular value in the diagonal matrix; Indicates the first The global LoRA projection parameter matrix obtained by reconstruction during rounds of federated training; The server analyzes the energy proportion of singular values ​​and extracts the (r+1)th to (r+s)th singular values ​​and their corresponding singular vectors to construct the residual term for the current round. ,in, This indicates the number of spectral components selected for constructing the residual term; This represents the residual matrix constructed in the current federated training round, used to characterize the residual terms that were not previously... The extraspectral components covered by each main spectral component; This represents the first singular value decomposition result of the global increment matrix. to A matrix consisting of left singular vectors; This represents the first singular value decomposition result of the global increment matrix. to A matrix consisting of right singular vectors; Indicates by the first to A diagonal matrix consisting of singular values; Representation matrix The transpose of the matrix; The updated residuals accumulated across rounds and the new global LoRA parameters are sent to the client for use in the next round of federated training.

2. The federated low-rank fine-tuning method for collaborative modeling of medical data according to claim 1, characterized in that, The singular value decomposition specifically includes: Generate random Gaussian matrices Its dimensions are ,in, Represents the global parameter increment matrix Column dimensions; Let represent the dimension of the random projected subspace, and Not less than the preset low-rank reconstruction rank value; Calculate intermediate variables ; For intermediate variables Perform QR matrix decomposition to obtain an orthogonal basis. ,Right now ; global increment matrix Map to a low-dimensional subspace constructed by Q and then perform SVD decomposition: Through calculation get The left singular value matrix is ​​obtained by decomposing the matrix into a complete matrix factorization. .

3. The federated low-rank fine-tuning method for collaborative modeling of medical data according to claim 1, characterized in that, forward The cumulative energy percentage based on the spectral components corresponding to singular values ​​is given by the following formula: In the formula, This represents the number of singular values ​​involved in the energy accumulation calculation, where ; This represents the first element of the global parameter increment matrix after singular value decomposition, arranged in descending order. One singular value; This represents the first element of the global parameter increment matrix after singular value decomposition, arranged in descending order. One singular value; Indicates the preset rank value used for low-rank reconstruction; This indicates the number of clients participating in the current federated training round; This represents the theoretical maximum effective upper bound of the global parameter increment matrix in the current federated training round.

4. The federated low-rank fine-tuning method for collaborative modeling of medical data according to claim 1, characterized in that, The client is a terminal device or a medical institution server, and the server is a central server or a cloud computing node.

5. A federated low-rank fine-tuning system for implementing the method of any one of claims 1-4, characterized in that, include: Multiple client devices, each client device including: The module includes a parameter receiving module, a residual fusion module, a low-rank fine-tuning module, a parameter increment generation module, and a parameter uploading module. Server nodes include: The system includes a parameter receiving module, a parameter aggregation module, a randomized singular value decomposition module, a low-rank reconstruction module, a residual extraction and caching module, and a parameter distribution module. Each client device is connected to the server node via a communication network, and the server node is configured to perform the method described in any one of claims 1 to 4.

6. An electronic device, characterized in that, include: One or more processors; Memory, which stores one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

Images