A multi-modal personalized federated learning method based on a hybrid expert adapter
By using a hybrid expert adapter approach, the adapter module is decoupled into private and shared parts. Combined with weighted aggregation of expert activation contribution rates, the problem of high computational and communication overhead in federated learning of multimodal large models is solved, achieving efficient personalized training and improved model generalization performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QILU UNIVERSITY OF TECHNOLOGY (SHANDONG ACADEMY OF SCIENCES)
- Filing Date
- 2026-02-04
- Publication Date
- 2026-04-28
AI Technical Summary
In federated learning of multimodal large models, existing technologies struggle to achieve efficient and personalized model training in heterogeneous data scenarios. Traditional methods suffer from high computational and communication overhead and poor model generalization performance.
A hybrid expert adapter-based approach is adopted, which decouples the adapter module into a private adapter and a shared hybrid expert adapter. The private adapter is kept locally for personalized feature extraction, while the shared adapter participates in global aggregation and performs weighted aggregation based on the expert activation contribution rate, thereby reducing communication and computation overhead and improving the model's generalization performance.
It enables efficient and personalized model training in heterogeneous data scenarios, reduces communication and computation overhead, improves the generalization performance and classification accuracy of the model, and takes into account the flexible combination of global general knowledge and local personalized knowledge.
Smart Images
Figure CN121638386B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal large model federated learning technology, specifically involving a multimodal personalized federated learning method based on a hybrid expert adapter. Background Technology
[0002] Pre-trained large-scale vision-language multimodal models have achieved breakthroughs in zero-shot reasoning and cross-modal tasks. However, the training of these large models heavily relies on massive amounts of data. In real-world applications such as finance, healthcare, and personal devices, data is often scattered across various clients in the form of data silos due to privacy protection and data ownership restrictions, making centralized training impossible. Federated learning, as a distributed machine learning paradigm, allows parties to collaboratively train models without sharing raw data, becoming a key technology for resolving the contradiction between data privacy and model intelligence.
[0003] In existing technologies, large multimodal models have a massive number of parameters. Performing full parameter fine-tuning and transmission in federated learning not only places extremely high demands on the client's computing power but also incurs unbearable communication overhead. Therefore, parameter efficient fine-tuning (PEFT) techniques such as Adapter and Prompt Tuning have been introduced into federated learning to adapt large models by training and transmitting only a small number of parameters. Traditional federated averaging algorithms train a general global model, but under highly heterogeneous data, the global model often struggles to adapt to the personalized needs of all clients, leading to a significant performance degradation on local tasks. To alleviate the problem of data heterogeneity, one approach is based on regularization, such as FedProx, which ensures convergence by constraining the local model from deviating too far from the global model, but this limits the model's ability to explore personalized features. Another approach is based on model decoupling, such as FedRep, which retains some layers for local training while aggregating the rest. However, in multimodal large model scenarios, simply personalizing the classification head is often insufficient to cope with complex modal differences; while if the feature extractor is simply averaged and aggregated, it is easy for the specific knowledge learned by different clients to interfere with each other, resulting in catastrophic forgetting. Summary of the Invention
[0004] The purpose of this invention is to address the aforementioned problems by providing a multimodal personalized federated learning method based on a hybrid expert adapter. This method involves freezing the CLIP backbone of a pre-trained multimodal large model and inserting a lightweight adapter module. This adapter module is decoupled into a private adapter and a shared hybrid expert MoE adapter. The private adapter remains local to preserve personalized features, while the shared MoE adapter participates in global aggregation to learn general knowledge. Furthermore, this invention proposes an aggregation strategy based on expert activation contribution rate, performing weighted aggregation according to the degree of expert usage by each client.
[0005] The method includes the following steps:
[0006] S1. The server and N clients respectively load the pre-trained vision-language CLIP model as the frozen backbone network, build the adapter module, and divide the adapter module into a shared part and a private part; the server initializes the shared adapter parameters, and each client initializes the private adapter parameters. After the initialization is completed, the server sends the initialized shared adapter parameters to all selected clients to complete the construction of the local model.
[0007] S2. Update the local model. In the t-th round of communication, each client receives the latest shared adapter parameters. The client overwrites the corresponding part in the local model with the received shared adapter parameters. The private adapter parameters inherit the client's previous training state.
[0008] S3. The client trains the updated local model using the local multimodal dataset, inputs image and text data, and after the CLIP model extracts features, it enters the adapter module. After the extracted features are processed by the private dimensionality reduction layer, the shared hybrid expert layer, and the private dimensionality increase layer, the contrast loss between image features and text features is calculated, the private adapter parameters and the shared adapter parameters are updated, and the cumulative frequency of each expert network in the shared hybrid expert layer being activated by the gated network is counted to generate an expert activation contribution statistical vector.
[0009] S4. The client uploads the updated shared adapter parameters and expert activation contribution statistics vector. The server performs hierarchical aggregation based on the information uploaded by the client. The gated network performs weighted averaging based on the amount of client data. The expert network performs weighted aggregation based on the amount of client data and usage rate. The server generates a new global shared model and distributes it to the next round of training clients.
[0010] S5. Repeat steps S2-S4 until the global shared model converges or the preset number of communication rounds is reached.
[0011] Furthermore, in step S1:
[0012] The shared adapter parameters This includes: the initial weights of the shared gating network and the shared expert network;
[0013] The private adapter parameters This includes the weights of the private dimensionality reduction projection layer and the dimensionality increase projection layer on both the text and image sides.
[0014] Furthermore, the specific steps of step S2 are as follows:
[0015] S21. In the At the start of round-robin communication, each client receives the latest shared adapter parameters from the server. ;
[0016] S22. The client overwrites the corresponding part in the local model with the received shared adapter parameters, represented as follows: Meanwhile, the private adapter parameters from the previous round of local training are retained unchanged, represented as ;
[0017] S23. If it is the first round of training, the private adapter parameters are randomly initialized; if it is a subsequent round, the private adapter parameters are inherited from the client's previous training state.
[0018] Furthermore, the specific steps of step S3 are as follows:
[0019] S31. The client iteratively trains the combined local model using the local multimodal dataset. Input image and text data, after feature extraction through the frozen CLIP backbone, enter the adapter module.
[0020] S32. Image data and text data are sequentially passed through a private dimensionality reduction layer, a shared hybrid expert layer, and a private dimensionality increase layer. The cross-entropy loss of image and text features is calculated, and the private adapter parameters and shared adapter parameters are updated simultaneously through backpropagation.
[0021] S33. During the forward propagation process of local training, the client records in real time the data of each expert network in the shared hybrid expert layer. The cumulative frequency of activation by the gating network is used to generate an expert activation contribution statistical vector. , indicating the first An expert on the client Contribution to the field.
[0022] Furthermore, the specific steps of step S32 are as follows:
[0023] S321. For small batches The first in Each sample, after being processed by the adapter module, yields enhanced image features. Through the text-side adapter C Categories Generate corresponding text features , corresponding to the True label of each sample The text features are denoted as ;
[0024] S322. Calculate the first... Image features of each sample Text features of all categories The cosine similarity is multiplied by the temperature coefficient. The predicted score is obtained;
[0025] S323. Use the Softmax function to transform Logits into a probability distribution and calculate its correlation with the true label. The cross-entropy loss between the two features is equivalent to maximizing the image features. Its correct category text features The similarity between features is calculated, while minimizing the similarity with features from other categories of text. The optimization objective is as follows: It updates both private adapter parameters and shared adapter parameters simultaneously through backpropagation.
[0026] Furthermore, the formula for calculating the predicted score is as follows:
[0027] ;
[0028] The optimization objective is: The calculation formula is:
[0029] .
[0030] Furthermore, the specific steps of step S33 are as follows:
[0031] S331. During the forward propagation of training, the client records in real time the cumulative contribution value of each expert network activated in the shared hybrid expert layer;
[0032] S332. The normalized weights of the gated network output are: For the first An expert Expert activation contribution statistics vector It is the sum of the contributions from the image stream and the text stream;
[0033] in, z Features of the shared hybrid expert layer are used as input.
[0034] Furthermore, the expert activation contribution statistical vector The calculation formula is:
[0035] ;
[0036] in, Representing image features for experts The allocation weights, Representing text features for experts The allocation weights, Reflecting on experts The importance of processing local multimodal data.
[0037] Furthermore, the specific steps of step S4 are as follows:
[0038] S41. The server collects the shared adapter parameters uploaded by each client. and expert activation contribution statistics vector ;
[0039] S42. The server adjusts the client dataset size accordingly. and Determine the aggregation weights and apply a standard weighted average based on the amount of data to the gated network:
[0040] ;
[0041] S43. For expert networks, dynamic perceptual aggregation is adopted, for the first... An expert The aggregation weight is determined by the amount of data and the first... K The usage rate of each expert is jointly determined:
[0042] ;
[0043] S44. After aggregation is complete, the server generates a new globally shared model. .
[0044] Furthermore, step S5 also includes the step of changing the rounds:
[0045] Perform server aggregation for the 0th time, and initialize the shared adapter parameters on the server. And then send it to the client to initialize the private adapter parameters on the image side. and text-side private adapter parameters ;
[0046] Perform the first local model update, sharing some updates. Private part inheritance ;
[0047] The first local model training was performed, using data to optimize the contrastive loss, updating all parameters, and statistical vectors were generated. After training, you will get ;
[0048] The first server aggregation is performed, and the client uploads the data. and The server aggregates the gating network and the expert network respectively, and obtains... ;
[0049] The training process is iterated until the globally shared model converges or reaches the preset number of communication rounds, at which point the training terminates.
[0050] As can be seen from the above technical solutions, the present invention has the following advantages:
[0051] First, by freezing the backbone parameters of the pre-trained multimodal large model CLIP, training is performed only on the lightweight adapter. Furthermore, the adapter is decoupled, requiring only the parameters of the shared hybrid expert part to be uploaded, while the private adapter parameters are kept locally. This not only avoids transmitting the massive backbone network parameters, but also eliminates the need to transmit the complete adapter network as in traditional methods. This reduces the requirements for edge device computing power and network bandwidth, reduces communication and computing overhead, and lowers the communication cost between the client and the server.
[0052] Secondly, by using an aggregation mechanism based on expert usage rate, the problem caused by data heterogeneity is alleviated. The aggregation weight is dynamically adjusted by using the expert activation frequency recorded during local training, so that the global model can perceive the data distribution preferences of different clients, thereby enabling the expert network to automatically align in function. This not only makes the global model have better generalization performance, but also, compared with simple average aggregation FedAvg, can better alleviate catastrophic forgetting or model drift caused by heterogeneous data.
[0053] Finally, this invention achieves a flexible combination and complementarity of global general knowledge and local personalized knowledge. The private adapter focuses on extracting specific features of local data, ensuring personalization capabilities while protecting privacy. The shared MoE adapter is responsible for retaining general knowledge, enabling the entire model to learn effective knowledge from other client models and complete local personalized tasks more efficiently, thereby improving the classification accuracy of multimodal models in non-independent and identically distributed scenarios. Attached Figure Description
[0054] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0055] Figure 1 This is a flowchart illustrating the multimodal personalized federated learning method based on a hybrid expert adapter, as described in this invention.
[0056] Figure 2 This is a schematic diagram of the multimodal personalized federated learning method based on a hybrid expert adapter in a specific embodiment of the present invention.
[0057] Figure 3 This figure shows the experimental results comparing the accuracy of different methods on the Caltech-101 dataset in the test of this invention.
[0058] Figure 4This figure shows the experimental results comparing the accuracy of different methods on the oxford_pets dataset in the test of this invention.
[0059] Figure 5 This figure shows the experimental results comparing the accuracy of different methods on the DTD dataset in the test of this invention. Detailed Implementation
[0060] Various embodiments of the invention will be described more fully in the specific steps of the multimodal personalized federated learning method based on a hybrid expert adapter, which will be described in detail below. The invention may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the invention to the specific embodiments disclosed herein, but rather the invention should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of the invention.
[0061] It should be understood that, when used in this specification, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.
[0062] The terms "one embodiment" or "some embodiments" used in this invention mean that one or more embodiments of the invention include the specific features, structures, or characteristics described in that embodiment. Therefore, the terms "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of the invention do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.
[0063] To make the objectives, features, and advantages of this invention more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solutions protected by this invention. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0064] Please see Figure 1 The diagram shows a flowchart of a multimodal personalized federated learning method based on a hybrid expert adapter. The method includes:
[0065] S1. The server and N clients respectively load the pre-trained vision-language CLIP model as the frozen backbone network, build the adapter module, and divide the adapter module into a shared part and a private part; the server initializes the shared adapter parameters, and each client initializes the private adapter parameters. After the initialization is completed, the server sends the initialized shared adapter parameters to all selected clients to complete the construction of the local model.
[0066] S2. Update the local model. In the t-th round of communication, each client receives the latest shared adapter parameters. The client overwrites the corresponding part in the local model with the received shared adapter parameters. The private adapter parameters inherit the client's previous training state.
[0067] S3. The client trains the updated local model using the local multimodal dataset, inputs image and text data, and after the CLIP model extracts features, it enters the adapter module. After the extracted features are processed by the private dimensionality reduction layer, the shared hybrid expert layer, and the private dimensionality increase layer, the contrast loss between image features and text features is calculated, the private adapter parameters and the shared adapter parameters are updated, and the cumulative frequency of each expert network in the shared hybrid expert layer being activated by the gated network is counted to generate an expert activation contribution statistical vector.
[0068] S4. The client uploads the updated shared adapter parameters and expert activation contribution statistics vector. The server performs hierarchical aggregation based on the information uploaded by the client. The gated network performs weighted averaging based on the amount of client data. The expert network performs weighted aggregation based on the amount of client data and usage rate. The server generates a new global shared model and distributes it to the next round of training clients.
[0069] S5. Repeat steps S2-S4 until the global shared model converges or the preset number of communication rounds is reached.
[0070] It should be noted that this embodiment divides the adapter module into shared and private parts, combined with a frozen CLIP pre-trained backbone, to ensure efficient global knowledge sharing while adapting to the local data characteristics of each client, enabling personalized modeling. Sharing a hybrid expert layer and expert activation contribution statistical vectors captures the adaptation value of different experts to local data. Combined with a hierarchical aggregation strategy, the gating network weights data based on volume, and the expert network aggregates data based on volume and usage rate, improving the generalization ability and adaptation accuracy of the global model. Contrast loss optimization and dual-adaptor parameter collaborative updates enhance image-text feature matching effects without requiring retraining of the backbone network, reducing training and communication overhead. Balancing model performance, personalized adaptation, and efficiency, this embodiment is suitable for various practical scenarios in multimodal federated learning.
[0071] As a refinement and extension of the specific implementation methods described above, for a complete explanation of the specific implementation process in this embodiment, please refer to [link / reference needed]. Figure 2 As shown, another multimodal personalized federated learning method based on a hybrid expert adapter is provided, which includes the following steps:
[0072] S1. The server and N clients respectively load the pre-trained vision-language CLIP model as the frozen backbone network, construct adapter modules, and divide the adapter modules into shared and private parts; the server initializes the shared adapter parameters, and each client initializes the private adapter parameters. After initialization, the server sends the initialized shared adapter parameters to all selected clients, completing the construction of the local model; in step S1:
[0073] The shared adapter parameters This includes: the initial weights of the shared gating network and the shared expert network;
[0074] The private adapter parameters This includes the weights of the private dimensionality reduction projection layer and the dimensionality increase projection layer on both the text and image sides.
[0075] It should be noted that, considering a multimodal federated learning scenario, and assuming that there are... There is one client and one server, and the client and server can communicate normally. This embodiment uses the first client as an example. The client, the first Taking local update training as an example, unless otherwise specified, the other clients and other local update training rounds are the same as the first round. The client, the first The same applies to local updates during training. The client has its own private dataset. The dataset contains pairs of multimodal data, each data point consisting of... It exists in the form of, in which For image data, This is the corresponding text description. Belongs to The mini-batch dataset allows the client to access both the local model and the latest received global model parameters. In this example, the model consists of a frozen, pre-trained CLIP model and a trainable adapter, defined as follows: For the adapter parameter set, such as the first The local adapter parameters obtained after training are updated locally in the first round, and the parameters obtained in the second round are updated locally. The global adapter parameters received before training are updated locally in each round.
[0076] S2. Update the local model. In the t-th round of communication, each client receives the latest shared adapter parameters. The client overwrites the corresponding part of the local model with the received shared adapter parameters, and the private adapter parameters inherit the client's previous training state. The specific steps of step S2 are as follows:
[0077] S21. In the At the start of round-robin communication, each client receives the latest shared adapter parameters from the server. ;
[0078] S22. The client overwrites the corresponding part in the local model with the received shared adapter parameters, represented as follows: Meanwhile, the private adapter parameters from the previous round of local training are retained unchanged, represented as ;
[0079] S23. If it is the first round of training, the private adapter parameters are randomly initialized; if it is a subsequent round, the private adapter parameters are inherited from the client's previous training state.
[0080] It should be noted that the client-side private components are maintained independently for the image side and the text side, with the image side having its own private adapter parameters. Including image dimensionality reduction projection layer and image upscaling projection layer :
[0081] ;
[0082] in, The image dimensionality reduction weight matrix, This is the corresponding bias vector. The image dimensionality-upgrading weight matrix, For the corresponding bias vector; similarly, for the text-side private adapter parameters. Includes text dimensionality reduction projection layer and text-level projection layer This part is responsible for mapping the heterogeneous features of each modality to a unified low-dimensional latent space while preserving the characteristics of local data distribution. The shared part consists of a shared hybrid expert layer, which includes a gating network. and A network of experts This part is shared by the image stream and the text stream, and is responsible for extracting common semantic features across modalities and clients.
[0083] For a piece of data Image features are extracted using CLIP. and text features The image stream is obtained through private image dimensionality reduction. Then input the shared hybrid expert adapter to get Finally, through private image upscaling and residual connection, the result is obtained. The text is obtained through text-private dimensionality reduction. Then input the same shared hybrid expert adapter to get Finally, through text-private dimensionality upscaling and residual join, the result is obtained. .
[0084] S3. The client trains the updated local model using the local multimodal dataset, inputs image and text data, and after the CLIP model extracts features, it enters the adapter module. The extracted features are processed by a private dimensionality reduction layer, a shared hybrid expert layer, and a private dimensionality increase layer. The contrast loss between image features and text features is calculated, the private adapter parameters and the shared adapter parameters are updated, and the cumulative frequency of each expert network in the shared hybrid expert layer being activated by the gated network is counted to generate an expert activation contribution statistical vector. The specific steps of step S3 are as follows:
[0085] S31. The client iteratively trains the combined local model using the local multimodal dataset. Input image and text data, after feature extraction through the frozen CLIP backbone, enter the adapter module.
[0086] S32. Image and text data are sequentially passed through a private dimensionality reduction layer, a shared hybrid expert layer, and a private dimensionality increase layer. The cross-entropy loss of image and text features is calculated, and the private adapter parameters and shared adapter parameters are updated simultaneously through backpropagation. The specific steps of step S32 are as follows:
[0087] S321. For small batches The first in Each sample, after being processed by the adapter module, yields enhanced image features. Through the text-side adapter C Categories Generate corresponding text features , corresponding to the True label of each sample The text features are denoted as ;
[0088] S322. Calculate the first... Image features of each sample Text features of all categories The cosine similarity is multiplied by the temperature coefficient. The predicted score is obtained; the formula for calculating the predicted score is:
[0089] ;
[0090] S323. Use the Softmax function to transform Logits into a probability distribution and calculate its correlation with the true label. The cross-entropy loss between the two features is equivalent to maximizing the image features. Its correct category text features The similarity between features is calculated, while minimizing the similarity with features from other categories of text. The optimization objective is as follows: It updates both private adapter parameters and shared adapter parameters simultaneously through backpropagation.
[0091] The optimization objective is: The calculation formula is:
[0092] .
[0093] S33. During the forward propagation process of local training, the client records in real time the data of each expert network in the shared hybrid expert layer. The cumulative frequency of activation by the gating network is used to generate an expert activation contribution statistical vector. , indicating the first An expert on the client Contribution to the field.
[0094] The specific steps of step S33 are as follows:
[0095] S331. During the forward propagation of training, the client records in real time the cumulative contribution value of each expert network activated in the shared hybrid expert layer;
[0096] S332. The normalized weights of the gated network output are: For the first An expert Expert activation contribution statistics vector It is the sum of the contributions from the image stream and the text stream;
[0097] Where z represents the features of the input-shared hybrid expert layer.
[0098] The expert activation contribution statistical vector The calculation formula is:
[0099] ;
[0100] in, Representing image features for experts The allocation weights, Representing text features for experts The allocation weights, Reflecting on experts The importance of processing local multimodal data.
[0101] S4. The client uploads the updated shared adapter parameters and expert activation contribution statistics vector. The server performs hierarchical aggregation based on the information uploaded by the client. The gating network performs weighted averaging based on the amount of client data, and the expert network performs weighted aggregation based on the amount of client data and usage rate. The server generates a new global shared model and distributes it to the clients for the next round of training. The specific steps of step S4 are as follows:
[0102] S41. The server collects the shared adapter parameters uploaded by each client. and expert activation contribution statistics vector ;
[0103] S42. The server adjusts the client dataset size accordingly. and Determine the aggregation weights and apply a standard weighted average based on the amount of data to the gated network:
[0104] ;
[0105] S43. For expert networks, dynamic perceptual aggregation is adopted, for the first... An expert The aggregation weight is determined by the amount of data and the first... K The usage rate of each expert is jointly determined:
[0106] ;
[0107] S44. After aggregation is complete, the server generates a new globally shared model. .
[0108] S5. Repeat steps S2-S4 until the global shared model converges or the preset number of communication rounds is reached.
[0109] Step S5 also includes the step of round change:
[0110] Perform server aggregation for the 0th time, and initialize the shared adapter parameters on the server. And then send it to the client to initialize the private adapter parameters on the image side. and text-side private adapter parameters ;
[0111] Perform the first local model update, sharing some updates. Private part inheritance ;
[0112] The first local model training was performed, using data to optimize the contrastive loss, updating all parameters, and statistical vectors were generated. After training, you will get ;
[0113] The first server aggregation is performed, and the client uploads the data. and The server aggregates the gating network and the expert network respectively, and obtains... ;
[0114] The training process is iterated until the globally shared model converges or reaches the preset number of communication rounds, at which point the training terminates.
[0115] In some embodiments, the method of the present invention is experimentally tested on three datasets: the DTD dataset, a dataset specifically for texture description and classification, containing 5640 images and 47 texture categories, used to verify the model's ability to capture low-level visual features and complex texture patterns; the Oxford_Pets dataset, a fine-grained image classification dataset containing 7349 images covering 37 pet categories; due to the small differences between different breeds, this dataset is used to verify the model's ability to recognize fine-grained features; and the Caltech-101 dataset, a classic general object recognition dataset containing 9144 images and 101 object categories, used to verify the model's generalization ability when facing diverse object shapes and general scenes.
[0116] The model used in this invention includes a pre-trained CLIP (ViT-B / 16) model. During training, the parameters of the CLIP image encoder and text encoder are kept frozen; the adapter module is designed as follows: the dimensionality reduction layer is a fully connected layer that compresses the input features to 32 dimensions and uses the ReLU activation function; the expert network consists of a 32-unit fully connected layer and a ReLU activation function; the dimensionality increase layer is a fully connected layer that restores the features from 32 dimensions to the original dimensions.
[0117] The number of shared experts is set to 4. The adapter module is inserted between layers 4-12 of the fixed encoder. The local iteration count is set to E=2, global communication is performed for 30 rounds, and mini-batch stochastic gradient descent (SGD) is used as the local optimizer. The data is partitioned using the Dirichlet distribution Dir. The degree of data heterogeneity is controlled by adjusting the Dir(β) parameter. The smaller the β, the higher the degree of heterogeneity.
[0118] To better examine the effectiveness of this invention, experiments were conducted in a resource-constrained environment: assuming 10 clients participated in federated learning, a 16-shot configuration was used, and the data heterogeneity rate β = 0.1, the server selected 100% of the clients to participate in the global aggregation. Experimental results on three datasets are as follows: Figure 3 , Figure 4 and Figure 5As shown, Ours represents the present invention, the comparison methods are FedPGP and PromptFL, the horizontal axis represents the number of communication rounds, and the vertical axis represents the accuracy. Figure 3 To compare the accuracy of different methods when using the Caltech-101 dataset, Figure 4 A comparison of the accuracy of different methods on the Oxford_Pets dataset. Figure 5 A comparison of the accuracy of different methods on the DTD dataset, by Figure 3 , Figure 4 , Figure 5 It can be seen that the present invention achieves 2-3% higher average accuracy of the model compared to existing methods compared to the baseline.
[0119] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0120] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A multimodal personalized federated learning method based on a hybrid expert adapter, characterized in that, The method includes the following steps: S1. The server and N clients respectively load the pre-trained vision-language CLIP model as the frozen backbone network, build the adapter module, and divide the adapter module into a shared part and a private part; the server initializes the shared adapter parameters, and each client initializes the private adapter parameters. After the initialization is completed, the server sends the initialized shared adapter parameters to all selected clients to complete the construction of the local model. S2. Update the local model. In the t-th round of communication, each client receives the latest shared adapter parameters. The client overwrites the corresponding part in the local model with the received shared adapter parameters. The private adapter parameters inherit the client's previous training state. S3. The client trains the updated local model using the local multimodal dataset, inputs image and text data, and after the CLIP model extracts features, it enters the adapter module. The extracted features are processed by a private dimensionality reduction layer, a shared hybrid expert layer, and a private dimensionality increase layer. The contrast loss between image features and text features is calculated, the private adapter parameters and the shared adapter parameters are updated, and the cumulative frequency of each expert network in the shared hybrid expert layer being activated by the gated network is counted to generate an expert activation contribution statistical vector. S4. The client uploads the updated shared adapter parameters and expert activation contribution statistics vector. The server performs hierarchical aggregation based on the information uploaded by the client. The gated network performs weighted averaging based on the amount of client data. The expert network performs weighted aggregation based on the amount of client data and usage rate. The server generates a new global shared model and distributes it to the client for the next round of training. S5. Repeat steps S2-S4 until the global shared model converges or the preset number of communication rounds is reached; The specific steps of step S4 are as follows: S41. The server collects the shared adapter parameters uploaded by each client. and expert activation contribution statistics vector ; S42. The server adjusts the size based on the client's dataset. and Determine the aggregation weights and apply a standard weighted average based on the amount of data to the gated network: ； Where N is the number of clients participating in the aggregation. This represents the number of samples in the i-th client's local dataset. This represents the gating parameters uploaded by the i-th client; S43. For expert networks, dynamic perceptual aggregation is adopted, for the first... An expert The aggregation weight is determined by the amount of data and the first... K The usage rate of each expert is jointly determined: ； in, This represents the number of samples in the i-th client's local dataset. This represents the weight of expert k in client i. This represents the network parameters of expert k in client i; S44. After aggregation is complete, the server generates a new globally shared model. .
2. The multimodal personalized federated learning method based on a hybrid expert adapter according to claim 1, characterized in that, In step S1: The shared adapter parameters This includes: the initial weights of the shared gating network and the shared expert network; The private adapter parameters This includes the weights of the private dimensionality reduction and dimensionality increase projection layers on both the text and image sides. Among them, the For image-side private adapter parameters, the These are text-side private adapter parameters.
3. The multimodal personalized federated learning method based on a hybrid expert adapter according to claim 1, characterized in that, The specific steps of step S2 are as follows: S21. In the At the start of round-robin communication, each client receives the latest shared adapter parameters from the server. ; S22. The client overwrites the corresponding part in the local model with the received shared adapter parameters, represented as follows: Meanwhile, the private adapter parameters from the previous round of local training are retained unchanged, represented as ; S23. If it is the first round of training, the private adapter parameters are randomly initialized. In subsequent rounds, the private adapter parameters are inherited from the client's previous training state.
4. The multimodal personalized federated learning method based on a hybrid expert adapter according to claim 1, characterized in that, The specific steps of step S3 are as follows: S31. The client iteratively trains the combined local model using the local multimodal dataset. Input image and text data, after feature extraction through the frozen CLIP backbone, enter the adapter module. S32. Image data and text data are sequentially passed through a private dimensionality reduction layer, a shared hybrid expert layer, and a private dimensionality increase layer. The cross-entropy loss of image and text features is calculated, and the private adapter parameters and shared adapter parameters are updated simultaneously through backpropagation. S33. During the forward propagation process of local training, the client records in real time the information of each expert network in the shared hybrid expert layer. The cumulative frequency of activation by the gating network is used to generate an expert activation contribution statistical vector. , indicating the first An expert on the client Contribution to the field.
5. The multimodal personalized federated learning method based on a hybrid expert adapter according to claim 4, characterized in that, The specific steps of step S32 are as follows: S321. For small batches The first in Each sample, after being processed by the adapter module, yields enhanced image features. Through the text-side adapter C Categories Generate corresponding text features , corresponding to the True label of each sample The text features are denoted as ; S322. Calculate the first... Image features of each sample Text features of all categories The cosine similarity is multiplied by the temperature coefficient. The predicted score is obtained; S323. Use the Softmax function to transform Logits into a probability distribution and calculate its correlation with the true label. The cross-entropy loss between the two features is equivalent to maximizing the image features. Its correct category text features The similarity between features is calculated, while minimizing the similarity with features from other categories of text. The optimization objective is as follows: It updates both private adapter parameters and shared adapter parameters simultaneously through backpropagation.
6. The multimodal personalized federated learning method based on a hybrid expert adapter according to claim 5, characterized in that, The formula for calculating the predicted score is: ； The optimization objective is: The calculation formula is: ； Among them, the This represents the batch size, i.e., the number of samples in the current batch.
7. The multimodal personalized federated learning method based on a hybrid expert adapter according to claim 4, characterized in that, The specific steps of step S33 are as follows: S331. During the forward propagation of training, the client records in real time the cumulative contribution value of each expert network activated in the shared hybrid expert layer; S332. The normalized weights of the gated network output are: For the first An expert Expert activation contribution statistics vector It is the sum of the contributions from the image stream and the text stream; in, z Features of the shared hybrid expert layer are used as input.
8. The multimodal personalized federated learning method based on a hybrid expert adapter according to claim 7, characterized in that, The expert activation contribution statistical vector The calculation formula is: ； in, Representing image features for experts The allocation weights, Representing text features for experts The allocation weights, Used to reflect experts The importance of processing local multimodal data.
9. The multimodal personalized federated learning method based on a hybrid expert adapter according to claim 1, characterized in that, Step S5 also includes the step of round change: Perform server aggregation for the 0th time, and initialize the shared adapter parameters on the server. And then send it to the client to initialize the private adapter parameters on the image side. and text-side private adapter parameters ; Perform the first local model update, sharing some updates. Private part inheritance ; The first local model training is performed, using data to optimize the contrastive loss, updating all parameters, and calculating the contribution statistics vector. After training, you will get , These are the parameters of the shared adapter obtained after training. These are the client-side private adapter parameters obtained after training. The first server aggregation is performed, and the client uploads the data. and contribution statistics vector The server aggregates the gating network and the expert network respectively, and obtains... ; The training process is iterated until the globally shared model converges or reaches the preset number of communication rounds, at which point the training terminates.
Citation Information
Patent Citations
Request processing method and device based on hybrid expert model, equipment and medium
CN120764690A
Dynamic federal mutual learning method and system for balancing personalization and generalization
CN121031721A