Large model federated learning method and system based on low-rank subspace dynamic clustering

Abstract

The application discloses a large model federated learning method and system based on low-rank subspace dynamic clustering. Each client of the application utilizes low-rank adaptation technology to perform efficient fine-tuning locally, and extracts a low-dimensional subspace of the parameter update quantity as a geometric representation through fast singular value decomposition; the server calculates the geometric similarity between the clients based on the subspace principal angle metric, constructs a bottom-up hierarchical clustering tree, and dynamically determines the optimal clustering structure by using the first-order difference of the link cost sequence; parameter aggregation is performed within the cluster to eliminate directional conflicts, and the intra-class aggregated model is used as the starting point for the next round of training, while a global model is introduced as a proximal constraint. While maintaining low communication bandwidth and computing cost, the application effectively aligns the optimization trajectory, eliminates the destructive interference of the gradient, and realizes the dual improvement of the generalization ability and personalized performance of the multi-modal large model in the heterogeneous data scene.

Description

Technical Field

[0001] This invention relates to a large-model federated learning method, which relates to the fields of artificial intelligence and federated learning technology, and specifically to a large-model federated learning method and system based on low-rank subspace dynamic clustering. Background Technology

[0002] With the rapid development of multimodal large language model technology, it has shown amazing potential in video understanding, behavior analysis, and intelligent interaction. For example, in the field of childcare, using intelligent systems to monitor and analyze infants' daily activities and potential dangerous behaviors in real time is of great significance for improving the quality of childcare.

[0003] However, traditional centralized training requires the necessary data to be aggregated in a single data center, creating a risk of data breaches. Infants' facial features, physical privacy, and daily behaviors constitute highly sensitive personal data, and parents and regulatory agencies often strictly prohibit the cloud transmission of such data, leading to a severe data silo effect. Federated learning, as a distributed machine learning paradigm, allows models to be collaboratively trained without data leaving the local machine, becoming a key technology for resolving these contradictions. However, in practical applications, it faces the following major technical problems and shortcomings: 1) The statistical heterogeneity of the data distribution is severe: different childcare institutions or home environments vary greatly. This non-independent and identically distributed characteristic of the data leads to divergence in the direction of model optimization on different clients.

[0004] 2) Performance degradation caused by conflicting optimization directions: In traditional federated aggregation, the server simply performs a weighted average of all client parameters. When the data distributions of different institutions differ significantly, the gradient updates uploaded by the clients may be orthogonal or even opposite in geometric direction, causing the global model to fail to converge to the optimal solution. This results in a significant drop in the recognition accuracy of the intelligent childcare system in specific scenarios.

[0005] 3) Communication and computation bottlenecks in large models: Existing clustering federated learning methods are mostly designed for small networks, requiring the transmission of all parameters or gradients to calculate similarity. However, for visual language models with billions of parameters, directly transmitting and calculating all parameters would cause unbearable communication bandwidth pressure and computational latency, failing to meet the real-time requirements of childcare monitoring systems.

[0006] Therefore, there is an urgent need for a federated fine-tuning method that can protect the privacy of infants and young children, effectively resolve optimization conflicts caused by data heterogeneity, and achieve high computational and communication efficiency. Summary of the Invention

[0007] To address the problems existing in the background technology, this invention provides a large-scale federated learning method and system based on low-rank subspace dynamic clustering. Addressing the issue that existing federated fine-tuning in non-independent and identically distributed data environments leads to conflicting optimization directions due to differences in client data distribution, resulting in parameter cancellation and model performance degradation during global aggregation, this invention proposes a novel solution applicable to childcare privacy data protection scenarios.

[0008] The technical solution adopted in this invention is: I. A large-scale federated learning system based on low-rank subspace dynamic clustering, comprising: The client update module obtains the model parameters and subspace matrix of the multimodal large language large model for each client based on the local private data of each client sharing the multimodal large language large model.

[0009] The orientation alignment module is optimized to obtain the geometric similarity between any two clients based on the subspace matrix of each client.

[0010] The dynamic clustering module includes a hierarchical clustering tree construction unit and a clustering structure calibration unit. The hierarchical clustering tree construction unit constructs a hierarchical clustering tree based on geometric similarity and then performs cluster merging. The clustering structure calibration unit determines several classes as the optimal clustering structure during the hierarchical cluster merging process.

[0011] The intra-class aggregation module is used to generate intra-class representative models for each class in the optimal clustering structure, and at the same time generate a global aggregated model based on the model parameters of the multimodal large language large model under each client.

[0012] The global constraint optimization module constructs constraints and loss functions during training iterations to perform the next round of local fine-tuning training on the multimodal large language model.

[0013] The client update module inputs the local private data of each client into the multimodal large language model for training. It uses the Low-Rank Adaptation (LoRA) method to decompose the update amount of the model weights of the multimodal large language model into a low-rank matrix as model parameters, and uses Fast Singular Value Decomposition (Fast SVD) to extract the subspace matrix of the update amount of the model weights. Then, it uploads the subspace matrix of each client and the update amount of the model weights of the multimodal large language model to the server.

[0014] In the hierarchical clustering tree construction unit of the dynamic clustering module, a hierarchical clustering tree is constructed based on the geometric similarity of each client using a hierarchical clustering method. Each class in the hierarchical clustering tree contains several clients of its own. The linkage cost between classes is obtained iteratively in a bottom-up manner. In each iteration, the two classes with the smallest linkage cost are merged.

[0015] In the clustering structure calibration unit of the dynamic clustering module, during the hierarchical clustering and merging process of each class in the hierarchical clustering tree, the minimum link cost of the iteration is obtained in real time and constructed as a link cost sequence. The first-order difference of the link cost sequence is obtained, and then the elbow method is used to determine the first peak position of the first-order difference as the cutoff point for stopping the hierarchical clustering and merging. The remaining classes are used as the optimal clustering structure for the current communication round.

[0016] In the intra-class aggregation module, the model parameters of each client in each class of the optimal clustering structure are weighted and aggregated by federated learning to generate an intra-class representative model for each class, which is then distributed to the corresponding client; the model parameters of each client are weighted and aggregated by federated learning to generate a global aggregated model.

[0017] In the global constraint optimization module, the client loads its own class representative model as initial parameters, uses the global aggregate model obtained in the previous iteration as a regularization constraint, and performs the next round of local fine-tuning training on the multimodal large language large model based on the local loss function containing the proximal constraint term.

[0018] II. A learning method for a large-scale federated learning system based on low-rank subspace dynamic clustering, including: Step S1: The client update module uses the local private data of each client sharing the multimodal large language model, low-rank adaptation method and fast singular value decomposition to obtain the model parameters and subspace matrix of the multimodal large language model under each client, and uploads the subspace matrix to the server.

[0019] Step S2: The optimization direction alignment module obtains the geometric similarity between each pair of clients based on the subspace matrix received by the server from each client.

[0020] Step S3: The dynamic clustering module constructs a bottom-up hierarchical clustering tree based on the geometric similarity obtained from the server and calculates the inter-class linkage cost. Then, it performs hierarchical cluster merging, dynamically determines the clustering cutoff point, completes the client's grouping, and determines several classes as the optimal clustering structure.

[0021] Step S4: Within each class of the optimal clustering structure, the intra-class aggregation module performs weighted aggregation of the model parameters of the intra-class clients to obtain the representative models of each class and then distributes them to the corresponding clients; at the same time, it generates a global aggregated model based on the model parameters of the multimodal large language large model under each client.

[0022] Step S5: During the training iteration, the global constraint optimization module loads the respective in-class representative models as initial parameters through the client, uses the global aggregate model obtained in the previous iteration as a regularization constraint, and performs the next round of local fine-tuning training on the multimodal large language large model based on the local loss function containing the proximal constraint term.

[0023] Each client in this invention utilizes low-rank adaptation technology for efficient local fine-tuning and extracts a low-dimensional subspace of parameter updates as a geometric representation through fast singular value decomposition. The server calculates the geometric similarity between clients based on the principal angle metric of the subspace, constructs a bottom-up hierarchical clustering tree, and dynamically determines the optimal clustering structure using the first-order difference of the link cost sequence. Parameter aggregation is performed within each cluster to eliminate directional conflicts, and the intra-cluster aggregation model is used as the starting point for the next round of training. A global model is also introduced as a proximal constraint. This invention effectively aligns the optimization trajectory and eliminates the destructive interference of gradients while maintaining low communication bandwidth and computational cost, achieving a dual improvement in the generalization ability and personalized performance of multimodal large models in heterogeneous data scenarios.

[0024] The beneficial effects of this invention are: 1. Significantly improve model performance under heterogeneous data: By using dynamic subspace clustering, clients with similar data distributions are grouped and aggregated, which effectively eliminates the conflict of gradient update directions and avoids the mutual cancellation of parameters, thus significantly improving the performance of the model in complex scenarios.

[0025] 2. Extremely low communication and computing costs: By utilizing low-rank adaptation and fast singular value decomposition techniques, similarity calculation can be completed by transmitting only a very low-dimensional subspace matrix, avoiding the transmission of all parameters, enabling large models with billions of parameters to run efficiently on resource-constrained edge devices.

[0026] 3. Balancing privacy protection and model generalization: The original data does not leave the local machine, strictly protecting data privacy; at the same time, the global near-end constraint mechanism ensures that the model can adapt to specific scenarios while still maintaining its ability to understand various general scenarios. Attached Figure Description

[0027] Figure 1 The present invention provides a flowchart of the overall system architecture. Figure 2 This is a flowchart of the client-side fast subspace extraction process used in this invention; Figure 3 This is a flowchart of the bottom-up hierarchical clustering method proposed in this invention. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] The large-scale federated learning system based on low-rank subspace dynamic clustering of this invention includes a client update module, an optimization direction alignment module, a dynamic clustering module, an intra-cluster aggregation module, and a global constraint optimization module. The client update module obtains the model parameters and subspace matrix of the multimodal large-scale language model for each client based on the local private data of each client sharing the multimodal large-scale language model. Specifically, the client update module inputs the local private data of each client into the multimodal large-scale language model for training. The local data is data from the same domain, such as medical, financial, or educational fields. The update amount of the model weights of the multimodal large-scale language model is decomposed into a low-rank matrix as model parameters using a low-rank adaptation method. The subspace matrix of the model weight update amount is extracted using fast singular value decomposition, serving as a geometric feature representing the client's optimization direction. Then, the subspace matrix of each client and the update amount of the model weights of the multimodal large-scale language model are uploaded to the server, thus forming a parameter direction pool and ensuring the privacy and security of the private data. The multimodal large-scale language model is a multimodal large-scale language model based on the LLaVA-1.5 architecture.

[0030] Low-rank fitting methods update the model weights. Represented as two low-rank matrices and The product of ,in, For the first i The amount of time the model weights are updated, the model weights , and They are respectively The number of rows and columns, , The rank of a low-rank decomposition is usually a value much smaller than 1. and Positive integers, i.e. .

[0031] After completing local training, without needing to reconstruct the full high-dimensional weight matrix, the left singular vector matrix of the updated model weights is directly calculated and extracted using fast singular value decomposition. As a subspace matrix; the client only considers the low-rank matrix and its corresponding subspace matrix. Uploaded to the server. The client only trains on local data and only uploads the updated parameters and corresponding subspace matrices after training to the server, ensuring the privacy and security of private data.

[0032] The optimization orientation alignment module obtains the geometric similarity between every two clients based on the subspace matrix of each client; within the optimization orientation alignment module, the subspace matrix of every two clients is... and The geometric similarity between client-optimized trajectories is measured based on the principal angles between each pair of subspace matrices, as detailed below: in, Indicates the first i The and the first j Geometric similarity between clients The dimension of the subspace matrix is ​​represented. This represents the Frobenius norm. This is a transpose.

[0033] The dynamic clustering module includes a hierarchical clustering tree construction unit and a clustering structure calibration unit. The hierarchical clustering tree construction unit constructs a hierarchical clustering tree based on the geometric similarity of each client and then performs cluster merging. Specifically, it uses a hierarchical clustering method to construct a hierarchical clustering tree based on the geometric similarity of each client. Each class in the hierarchical clustering tree contains several clients of its own. It uses a bottom-up approach to iteratively obtain the link cost between classes. In each iteration, the two classes with the smallest link cost are merged.

[0034] Linkage cost The definition is a unit value of 1 and two classes. and The difference in average similarity among all cross-class client pairs is as follows: in, and Representing classes respectively and class The number of clients included.

[0035] The clustering structure calibration unit of the dynamic clustering module determines several classes as the optimal clustering structure during the hierarchical clustering merging process. Specifically, during the hierarchical clustering merging process for each class in the hierarchical clustering tree, the minimum link cost of the iteration is obtained in real time and constructed as a link cost sequence. The first-order difference of the link cost sequence is obtained, and then the elbow rule is used to determine the first peak position of the first-order difference as the cutoff point to stop the hierarchical clustering merging. The remaining classes are then used as the optimal clustering structure for the current communication round. The elbow rule selects the first-order difference value. The location of the earliest peak is used as the cutoff point for stopping merging, thereby dynamically determining the current number of clusters and the partitioning structure, preventing geometrically conflicting clusters from being merged. The intra-class aggregation module is used to generate intra-class representative models for each class in the optimal clustering structure, and simultaneously generate a global aggregation model based on the model parameters of the multimodal large language large model under each client. Specifically, the model parameters of each client in each class in the optimal clustering structure are weighted and aggregated by federated learning to generate an intra-class representative model for each class, which is then distributed to the corresponding client to replace the traditional global average aggregation model as the initialization parameters for the next local training of the client in that class; the model parameters of each client are weighted and aggregated by federated learning to generate a global aggregation model.

[0036] During the training iteration, the global constraint optimization module constructs constraints and loss functions to perform the next round of local fine-tuning training on the multimodal large language model. Specifically, each client loads its own in-class representative model as initial parameters, uses the global aggregate model obtained in the previous iteration as a regularization constraint, and performs the next round of local fine-tuning training on the multimodal large language model based on the local loss function containing proximal constraints, limiting the client's local optimization trajectory to not deviate from the global optimal solution space. The large model training is completed when all clients' local private data have been trained or the preset number of iterations has been reached.

[0037] Local loss function including proximal constraint terms Specifically as follows: in, The regularization coefficient is . These are the model parameters for the current client. These are the parameters of the previous round's global aggregation model. t This represents the current iteration round.

[0038] The learning method of the large-model federated learning system based on low-rank subspace dynamic clustering of the present invention is as follows: Step S1: The client update module uses the local private data of each client sharing the multimodal large language model, low-rank adaptation method and fast singular value decomposition to obtain the model parameters and subspace matrix of the multimodal large language model under each client, and uploads the subspace matrix to the server.

[0039] Step S2: The optimization direction alignment module obtains the geometric similarity between each pair of clients based on the subspace matrix received by the server from each client.

[0040] Step S3: The dynamic clustering module constructs a bottom-up hierarchical clustering tree based on the geometric similarity obtained from the server and calculates the inter-class linkage cost. Then, it performs hierarchical cluster merging, dynamically determines the clustering cutoff point, completes the client's grouping, and determines several classes as the optimal clustering structure.

[0041] Step S4: Within each class of the optimal clustering structure, the intra-class aggregation module performs weighted aggregation of the model parameters of the intra-class clients to obtain the representative models of each class and then distributes them to the corresponding clients; at the same time, it generates a global aggregated model based on the model parameters of the multimodal large language large model under each client.

[0042] Step S5: During the training iteration, the global constraint optimization module loads the respective in-class representative models as initial parameters through the client, uses the global aggregate model obtained in the previous iteration as a regularization constraint, and performs the next round of local fine-tuning training on the multimodal large language large model based on the local loss function containing the proximal constraint term.

[0043] Embodiments of this invention construct a cross-institutional intelligent monitoring federated learning system for infants and toddlers. For example... Figure 1 As shown, the system mainly consists of a server and clients distributed across different childcare institutions. The system aims to leverage heterogeneous video data distributed across various locations to collaboratively fine-tune a high-performance multimodal large language model. The specific workflow of this embodiment is as follows: like Figure 1 The diagram shows the complete closed loop from (1) to (3), where (1) client-side local training update and corresponding subspace extraction, (2) server-side dynamic clustering and optimization direction clustering alignment, and (3) intra-class aggregation and distribution, as detailed below: 1. Client-side local updates and lightweight feature extraction: such as Figure 1 As shown, in the communication rounds Each childcare institution's client machine fine-tunes the model based on its local private monitoring dataset. Since fine-tuning all parameters of a large model is too costly, this system employs a low-rank adaptation technique, freezing the backbone parameters of the pre-trained large model and updating only the low-rank matrix. To avoid transmitting all parameters or a massive update matrix to the server, the client performs fast singular value decomposition locally, such as... Figure 2 As shown, the principal subspace matrix of parameter update is extracted using fast singular value decomposition. Specifically: First, for the matrix Perform QR factorization to obtain an orthogonal matrix. and upper triangular matrix Next, calculate the intermediate matrix. and the intermediate matrix Perform singular value decomposition to obtain its left singular vector. Finally, through calculation Restore the main left singular vector matrix of the original update. And extract the matrix. The former Dimension as subspace This indicates uploading.

[0044] 2. Optimize orientation alignment and dynamic clustering: After receiving the subspace vectors uploaded by each client, the server calculates the geometric similarity between any two client subspaces. If the data distributions of two clients are similar, the subspace overlap of their update directions is high, and the similarity is closer to 1.

[0045] 3. Dynamic hierarchical clustering: The server calculates the link cost in real time as the cost for aggregating two classes, such as... Figure 3 As shown, this constructs a bottom-up hierarchical clustering tree. (See diagram) Figure 1 The system automatically identifies the inflection point in the "linkage cost-merging step" curve. At the inflection point, the marginal increment of the link cost is the largest, meaning that continuing to merge would forcibly merge two groups with conflicting optimization directions. The system then severs the clustering tree at this point, dynamically generating several classes.

[0046] 4. Intra-class aggregation and dispatch, such as Figure 1 As shown, within the divided classes, the server performs weighted aggregation of model parameters. Specifically, the weighted aggregation of model parameters of clients within a class is performed by calculating a weighting coefficient based on the ratio of the size of the client's local dataset to the total size of all client datasets within the class, and then summing the model parameters of each client within the class based on this weighting coefficient to generate a representative model for the class.

[0047] 5. After receiving the intra-class model, the local training client with global constraints introduces global proximal constraints in the next round of training to ensure that the model does not lose its ability to recognize general features while adapting to the local scene.

[0048] The above specific embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A large-scale federated learning system based on low-rank subspace dynamic clustering, characterized in that, include: The client update module, based on the local data of each client sharing the multimodal large language large model, obtains the model parameters and subspace matrix of the multimodal large language large model under each client; The orientation alignment module is optimized to obtain the geometric similarity between any two clients based on the subspace matrix of each client. The dynamic clustering module includes a hierarchical clustering tree construction unit and a clustering structure calibration unit. The hierarchical clustering tree construction unit constructs a hierarchical clustering tree based on each geometric similarity and then performs cluster merging. The clustering structure calibration unit determines several classes as the optimal clustering structure during the hierarchical cluster merging process. The intra-class aggregation module is used to generate intra-class representative models for each class in the optimal clustering structure, and at the same time generate a global aggregation model based on the model parameters of the multimodal large language large model under each client. The global constraint optimization module constructs constraints and loss functions during training iterations to perform the next round of local fine-tuning training on the multimodal large language model.

2. The large-scale federated learning system based on low-rank subspace dynamic clustering according to claim 1, characterized in that: The client update module inputs the local data of each client into the multimodal large language model for training. It uses the low-rank adaptation method to decompose the update amount of the model weights of the multimodal large language model into a low-rank matrix as model parameters, and uses fast singular value decomposition to extract the subspace matrix of the update amount of the model weights. Then, it uploads the subspace matrix of each client and the update amount of the model weights of the multimodal large language model to the server.

3. The large-scale federated learning system based on low-rank subspace dynamic clustering according to claim 1, characterized in that: In the hierarchical clustering tree construction unit of the dynamic clustering module, a hierarchical clustering tree is constructed based on the geometric similarity of each client using a hierarchical clustering method. Each class in the hierarchical clustering tree contains several clients of its own. The linkage cost between classes is obtained iteratively in a bottom-up manner. In each iteration, the two classes with the smallest linkage cost are merged.

4. The large-scale federated learning system based on low-rank subspace dynamic clustering according to claim 3, characterized in that: In the clustering structure calibration unit of the dynamic clustering module, during the hierarchical clustering and merging process of each class in the hierarchical clustering tree, the minimum link cost of the iteration is obtained in real time and constructed as a link cost sequence. The first-order difference of the link cost sequence is obtained, and then the elbow rule is used to determine the first peak position of the first-order difference as the cutoff point to stop the hierarchical clustering and merging. The remaining classes are taken as the optimal clustering structure.

5. The large-scale federated learning system based on low-rank subspace dynamic clustering according to claim 1, characterized in that: In the intra-class aggregation module, the model parameters of each client in each class of the optimal clustering structure are weighted and aggregated by federated learning to generate an intra-class representative model for each class, which is then distributed to the corresponding client; the model parameters of each client are weighted and aggregated by federated learning to generate a global aggregated model.

6. The large-scale federated learning system based on low-rank subspace dynamic clustering according to claim 1, characterized in that: In the global constraint optimization module, the client loads its own class representative model as initial parameters, uses the global aggregate model obtained in the previous iteration as a regularization constraint, and performs the next round of local fine-tuning training on the multimodal large language large model based on the local loss function containing the proximal constraint term.

7. The learning method for a large-scale federated learning system based on low-rank subspace dynamic clustering according to any one of claims 1-6, characterized in that, include: Step S1: The client update module uses the local data of each client sharing the multimodal large language model, low-rank adaptation method and fast singular value decomposition to obtain the model parameters and subspace matrix of the multimodal large language model under each client, and uploads the subspace matrix to the server. Step S2: The optimization direction alignment module obtains the geometric similarity between every two clients based on the subspace matrix received by the server from each client; Step S3: The dynamic clustering module constructs a bottom-up hierarchical clustering tree based on the geometric similarity obtained from the server and calculates the inter-class linkage cost. Then, it performs hierarchical cluster merging, dynamically determines the clustering cutoff point, completes the client's grouping, and determines several classes as the optimal clustering structure. Step S4: Within each class of the optimal clustering structure, the intra-class aggregation module performs weighted aggregation of the model parameters of the intra-class clients to obtain the representative models of each class and then distributes them to the corresponding clients; at the same time, it generates a global aggregated model based on the model parameters of the multimodal large language large model under each client. Step S5: During the training iteration, the global constraint optimization module loads the respective in-class representative models as initial parameters through the client, uses the global aggregate model obtained in the previous iteration as a regularization constraint, and performs the next round of local fine-tuning training on the multimodal large language large model based on the local loss function containing the proximal constraint term.

Images