Federal multi-modal large model adaptation method based on modal complementary cooperation

Abstract

The application discloses a federal multi-modal large model adaptation method based on modal complementation cooperation, which comprises the following steps: S1 initialization preparation; S2 grouping and issuing; S3 client local training; S4 uploading; S5 server side teacher signal aggregation; and S6 server side complementary contribution evaluation and aggregation. The method can reduce negative migration, reduce aggregation pollution, stably and controllably realize cross-modal knowledge transmission, has low communication and end side cost, and is suitable for various basic models and tasks.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and federated learning technology, specifically a federated multimodal large model adaptation method based on modal complementarity and collaboration. Background Technology

[0002] With the widespread application of multimodal large models in fields such as finance, healthcare, and education, the problem of missing modalities is often exposed in specific engineering implementations. Due to limitations in data acquisition and input devices, as well as the lack of available modal data, the training effect of multimodal large models is severely limited, and the lack of modalities leads to a significant decline in training performance. In federated learning scenarios, due to privacy protection requirements, the available multimodal data for training is even more strictly limited, making the problem of missing modalities even more prominent in the training of multimodal large models in federated scenarios.

[0003] Some existing methods employ "generative completion of missing modalities," which can easily introduce data illusions and privacy risks. Furthermore, in federated scenarios, it is difficult to uniformly verify the completion quality, making it highly susceptible to noise. Mainstream federated VLM / MLLM adaptation methods (such as ICCV 2025 FedMVP and ICLR 2025 pFedMoAP) default to "modal completeness" and do not model "modal availability" as a core variable, lacking structured handling of semantic subspace inconsistencies. ICCV 2025 FedPrime only focuses on "prompt instruction alignment and clustering aggregation," failing to transform missing modalities into complementary advantages and not addressing the problem that "cross-modal alignment signals from a few IT clients are drowned out by a majority of single-modal clients."

[0004] The existing technology has the following shortcomings:

[0005] The model parameter updates of clients under different modal conditions have subspace biases, and direct aggregation is prone to semantic drift and negative transfer (the quality of the dominant knowledge in the aggregation is poor, which contaminates the model training).

[0006] When missing modality completion relies on generative models, it is difficult to uniformly measure and verify the completion quality, which can easily amplify noise and expose privacy.

[0007] Cross-modal alignment signals provided by paired text-to-image (IT) clients are easily diluted when the proportion is small. Ordinary weighting based on data volume / loss improvement can lead to insufficient attention to cross-modal anchors, resulting in drift of the semantic coordinate system learned by I-only / T-only models. This ultimately manifests as negative transfer / aggregation pollution, making it difficult for the global model to establish stable cross-modal anchors.

[0008] The aforementioned shortcomings lead to problems such as semantic drift, aggregation contamination, or alignment signal dilution in the training of federated multimodal models with mixed modalities in existing technologies. Summary of the Invention

[0009] The purpose of this invention is to address the semantic subspace inconsistency, aggregation pollution, and negative transfer problems caused by inconsistent client modality availability (I-only, T-only, and IT hybrids, which may be accompanied by sample-level missing data) in federated learning scenarios. This invention proposes a federated multimodal large model adaptation method based on modal complementarity and collaboration. This method reduces negative transfer, minimizes aggregation pollution, enables stable and controllable cross-modal knowledge propagation, has low communication and client-side costs, and is applicable to various basic models and tasks.

[0010] The technical solution to achieve the objective of this invention is:

[0011] A federated multimodal large model adaptation method based on modal complementarity and cooperation, comprising a system consisting of a server and multiple clients, the method comprising the following steps:

[0012] S1 Initialization Preparation: The client first performs modal masking on the local dataset, modeling the modal availability of the data as a first-order variable, namely the modal availability mask m. m identifies images as I, text as T, and text and images as IT. The client determines the dominant mask based on the data it holds locally. That is, clients that only hold image data are I-only clients, clients that only hold text data are T-only clients, and clients that hold paired datasets of images and text are IT clients. The client's dominant mask is backed up on the server side. Based on the client's dominant mask, the server distributes the initial basic model that conforms to the client's characteristics to all clients participating in federated learning.

[0013] The local dataset after being partitioned using modal masks is in the following form: ,

[0014] in and These represent paired image and text data, respectively. If a modality is missing, then... ;

[0015] The label represents the sample in the dataset, that is, the semantic category to which the sample belongs;

[0016] S2 grouping and distribution:

[0017] The server selects the client set to participate in round t. The server sends data to the participating client set. Each client in the process sends out adaptation parameters;

[0018] S3 client local training:

[0019] The client freezes the main parameters of the base model. Under these conditions, only update the PEFT parameter in the adaptation parameters;

[0020] The PEFT parameters are injected into the basic model in the form of residuals: for the output representation of any modal tower, let the frozen forward be... Then the equivalent forward representation after PEFT injection is:

[0021] + +

[0022] in This represents a lightweight residual branch implemented using PEFT parameters. Selection is triggered by the modal availability mask;

[0023] Let the frozen visual tower be The frozen text tower is , respectively and Do Normalization ,get:

[0024] Image representation:

[0025]

[0026] Text representation:

[0027]

[0028] Optional gating fusion can be added to the IT client:

[0029]

[0030] The representations obtained by each client are then passed forward for loss calculation;

[0031] After this round of training, the IT client calculates the class prototype and statistics (variance and class sample count) of each sample class c in the local dataset used for training as the teacher package:

[0032]

[0033]

[0034] Compared to single-modal clients, the IT client outputs updated shared dock module parameters and related modal expert parameters, as well as teacher package statistics. ;

[0035] S4 upload:

[0036] The client-side training in this round was completed locally and then uploaded to the server:

[0037] Shared dock update ;

[0038] Corresponding modal expert update ;

[0039] Contribution assessment scalar;

[0040] Additional Teacher Package Upload Statistics from IT Clients ;

[0041] The uploaded data consists of parameter increments, gradients, or statistics, not the original data.

[0042] S5 server-side teacher signal aggregation:

[0043] The server aggregates the class prototypes uploaded by IT clients to form a global teacher prototype:

[0044]

[0045] And maintain the variance statistics corresponding to the class prototype. :

[0046]

[0047] when Below the threshold This indicates that the number of samples for the corresponding class in the client's local dataset is small, the provided class prototype is not representative, and it is not included in the aggregation to improve reliability;

[0048] The server updates the global model on the server by aggregating the model updates uploaded by the clients participating in this round of training. The updated global model is then displayed in the anchor set. Obtain logits Used for teacher distillation, serving as a global teacher signal;

[0049] S6 Server-Side Complementary Contribution Assessment and Aggregation:

[0050] The server calculates the complementary contribution scores of the clients. Normalize the relevant indices within each mode group and then perform a linear combination:

[0051]

[0052] in:

[0053] Indicates alignment gain;

[0054] Indicates teacher quality;

[0055] Indicates stability;

[0056] Then, softmax is used within each modality group to obtain the group weights:

[0057]

[0058] This is the reciprocal form of the weighted temperature hyperparameter;

[0059] And by introducing cross-group quotas (anchor protection), cross-group aggregate weights are obtained:

[0060]

[0061] First, competition within the group (softmax), then cross-group quotas ( This is to simultaneously address the issues of "intra-group quality disparity" and "inter-group IT dilution"; ordinary single-layer weighted averages cannot simultaneously satisfy the requirement of servers separately aggregating shared base parameters and expert parameters.

[0062] expert Intra-group aggregation:

[0063]

[0064] Shared dock Use cross-group weights polymerization:

[0065]

[0066] After completion, proceed to the next training round t+1;

[0067] If the preset number of iterations has been completed or the training performance of the models on the local dataset for each client has converged, then the federated learning phase is considered complete.

[0068] After the federated learning phase is completed, each client uses the iterated model to perform model adaptation for the same task on the new data.

[0069] The adaptation parameters in step S2 include the following:

[0070] The server distributes shared docking parameters to all clients in this round. ;

[0071] The server, based on the dominant mask backup of each client on the server, sends model parameters that match the characteristics of each client. If the client's dominant mask is I-only, then image expert parameters are also sent. If the client-side master mask is T-only, then additional text expert parameters will be sent. If the client-dominant mask is IT, then the IT expert parameters will be sent simultaneously with the image and text expert parameters. and optional gating parameters ;

[0072] The server distributes the teacher package statistics for the current round and the global IT prototype aggregated from the previous round in each round. In public anchorage The logits soft label (which can be compressed into top-k probabilities and indices) on the top-k list.

[0073] The representation obtained in step S3 is further passed forward for loss calculation, for I-only clients:

[0074] Calculate the task loss for each batch B. :

[0075]

[0076] Furthermore, class prototypes and statistics are introduced as teacher signals to calculate the alignment complementarity loss:

[0077]

[0078] Hyperparameters used to maintain numerical stability are typically taken to be extremely small.

[0079] in For image encoding representation, logits distillation can be optionally performed on the anchor set batch b. This indicates the corresponding mode in the anchor set batch. The logits obtained from the above;

[0080]

[0081]

[0082] That is, anchor set Sample data on;

[0083] The temperature hyperparameter is used to adjust the output distribution characteristics of softmax. Its value is between 0.1 and 0.5. The smaller the hyperparameter, the sharper the distribution and the greater the probability difference; the larger the hyperparameter, the smoother the distribution and the smaller the probability difference.

[0084] Final total loss:

[0085]

[0086] , All are single-modal loss weights, which are manually set according to specific training needs;

[0087] The I-only client model updates relevant modal expert parameters, shared base parameters, and optional task headers in reverse based on the loss.

[0088] The teacher signal used for alignment is the global teacher signal.

[0089] The representation obtained in step S3 is further passed forward for loss calculation, for T-only clients:

[0090] Similar to the loss calculation process for I-only clients, the representation is replaced with a text representation. and align with the same teacher prototype ,renew ;

[0091] The representation obtained in step S3 is further passed forward for loss calculation, for IT clients:

[0092] Calculate the task loss for each batch B. :

[0093]

[0094] Simultaneously optimize task loss and cross-modal consistency loss on paired data:

[0095]

[0096] Final total loss:

[0097]

[0098] The full-modal loss weights are manually set according to specific training needs.

[0099] The IT client model updates relevant modal expert parameters, shared base parameters, and optional task heads in reverse order based on the loss.

[0100] This technical solution transforms the lack of modality from a problem into a complementary signal source, and suppresses semantic drift through complementary distillation and representation alignment. At the same time, it avoids the noise and illusions introduced by generative completion, and achieves robust training of federated multimodal large models under mixed lack of modality.

[0101] Without performing generative completion, robust adaptation of federated multimodal large models is achieved, and cross-modal alignment knowledge of a small number of IT clients can be stably propagated.

[0102] This technical solution uses the modal availability mask m as a primary variable to decompose the trainable adaptation parameters into a "shared base + modal condition expert" and selectively distributes / returns them according to the mask, thereby reducing mutual pollution of cross-modal updates.

[0103] Simultaneously, IT clients are treated as "statistical cross-modal teachers," generating and uploading category-level prototypes, variances, and counts, which are then aggregated by the server to form a global teacher, providing a unified semantic anchor for clients lacking modality; teacher statistics may also include, but are not limited to (optional implementation):

[0104] Second-order moments / covariances (intra-class covariance, cross-modal covariance) are used for finer alignment;

[0105] Inter-class boundary statistics (class center distance, margin statistics);

[0106] Compressed forms of teacher signals (quantized, sparse, top-k, etc.).

[0107] The server-side is based on complementary contribution scores. Perform grouped weighted aggregation and apply cross-group quotas and lower bounds for IT weights. ≥ Anchor point protection prevents the alignment signal from being diluted due to the small proportion of IT clients; anchor point protection is not limited to this one implementation, but rather a class of equivalent implementations (optional implementations):

[0108] Lower bound races: ≥ ,or ≥ (Lower bound of total IT group weight) or set a lower bound for IT contribution to shared base θs.

[0109] Upper bound / Squeezed group: Simultaneously set an upper bound for I / T-only to form an equivalent expression of "ensuring that IT is not too small".

[0110] Projection / Lagrange implementation: First calculate the unconstrained weights, then perform the projection to satisfy the constraints.

[0111] dynamic : It can adapt to changes in IT usage percentage, teacher quality, and degree of migration.

[0112] Beneficial effects:

[0113] Reduce negative transfer: Provide a unified semantic anchor to the modality-deficient client through IT teacher prototypes / soft tags to reduce semantic drift of I-only / T-only and reduce alignment loss on the common anchor set A;

[0114] Reduce aggregation pollution: Modal masking experts isolate updates under different modal conditions within their respective expert groups, while preserving knowledge transfer from the shared foundation;

[0115] Stable and controllable cross-modal knowledge propagation: Anchor point protection and constraints are achieved through complementary contribution scoring and IT quota / weight lower bounds. ,and (Total quota) to prevent IT signals from being diluted;

[0116] Low communication and end-side costs: The preferred approach is to freeze the basic model and update only PEFT and a small amount of statistical information; optional logits teacher signals can be distributed with top-k probabilities and indices, with the prototype being class-level statistics;

[0117] It is applicable to a variety of basic models and tasks: it can be extended to VLM / MLLM and tasks such as classification, retrieval, alignment, and instruction fine-tuning, as long as the representation or output distribution can be defined. Attached Figure Description

[0118] Figure 1 This is a training flowchart for an example;

[0119] Figure 2 This is a schematic diagram of the system framework in the embodiment;

[0120] Figure 3 This is a schematic diagram of the client-side local training process in the embodiment;

[0121] Figure 4 This is a schematic diagram of the server-side aggregation process between rounds in the embodiment;

[0122] Figure 5 This is a schematic diagram of the server-side aggregation weight calculation process in the embodiment. Detailed Implementation

[0123] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0124] Example:

[0125] Taking medical enterprises as an example, due to the different data collection equipment, diagnostic processes, and backup status of diagnostic data among various institutions, the modal differences and data volume of medical data in some fields are large. IT nodes may be a very small number of institutions with relatively small data volumes. At the same time, due to the strict privacy requirements of the medical industry, the method in this example is suitable for joint training and learning under the condition of privacy protection.

[0126] Taking medical data for a lung disease as an example, the image modality corresponds to lung films, the text modality corresponds to diagnostic opinions, and the tags identify semantic category information such as the infection status and severity of the disease. It is set that there is a single central server and K client nodes participating in the federated learning system. The client nodes correspond to medical institutions. The IT modality has a very small percentage of complete data, with only a few IT nodes having complete paired data.

[0127] The CLIP model was used for implementation, the PEFT parameter update method was implemented in LoRA form, and the task head was selected to implement the classification task.

[0128] Reference Figure 1 , Figure 2 A federated multimodal large model adaptation method based on modal complementarity and cooperation, comprising a system consisting of a server and multiple clients, the method comprising the following steps:

[0129] S1 Initialization Preparation: The client first performs modal masking on the local dataset, modeling the modal availability of the data as a first-order variable, namely the modal availability mask m, where m identifies images as I, text as T, and text and images as IT. The client then determines the dominant mask based on the data it holds: clients holding only image data are I-only clients, clients holding only text data are T-only clients, and clients holding paired datasets of images and text are IT clients. The client's dominant mask is backed up on the server. Based on the client's dominant mask, the server distributes an initial basic model that matches the client's characteristics to all clients participating in federated learning.

[0130] The local dataset after being partitioned using modal masks is in the following form: ,

[0131] in and These represent paired image and text data, respectively. If a modality is missing, then... ;

[0132] The label represents the sample in the dataset, that is, the semantic category to which the sample belongs;

[0133] S2 grouping and distribution:

[0134] The server selects the client set to participate in round t. The server sends data to the participating client set. Each client in the process sends out adaptation parameters:

[0135] Shared docking station adaptation parameters will be distributed to all clients in this round. ;

[0136] If the client-dominant mask is I-only, then additional image expert parameters will be sent. If the client-side master mask is T-only, then additional text expert parameters will be sent. If the client-dominant mask is IT, then the IT expert parameters will be sent simultaneously with the image and text expert parameters. and optional gating parameters ;

[0137] The server distributes the teacher package statistics for the current round and the global IT prototype aggregated from the previous round in each round. In public anchorage The logits soft label (which can be compressed into top-k probabilities and indices);

[0138] S3 client local training, such as Figure 3 As shown:

[0139] The client freezes the main parameters of the base model. Under these conditions, only the PEFT parameters are updated, i.e., the shared docking adapter parameters. With corresponding modal expert parameters and optional task headers;

[0140] The PEFT parameters are injected into the basic model in the form of residuals: for the output representation of any modal tower, let the frozen forward be... Then the equivalent forward representation after PEFT injection is:

[0141] + +

[0142] in This represents a lightweight residual branch implemented using PEFT parameters. Selection is triggered by the modal availability mask;

[0143] Let the frozen visual tower be The frozen text tower is , respectively and Do Normalization ,get:

[0144] Image representation:

[0145]

[0146] Text representation:

[0147]

[0148] Optional gating fusion can be added to the IT client:

[0149]

[0150] The obtained representation is then passed forward for loss calculation;

[0151] Loss calculation for I-only clients:

[0152] Calculate the task loss for each batch B. :

[0153]

[0154] Furthermore, class prototypes and statistics are introduced as teacher signals to calculate the alignment complementarity loss:

[0155]

[0156] Hyperparameters used to maintain numerical stability;

[0157] in For image encoding representation, logits distillation can be optionally performed on the anchor set batch b. This indicates the corresponding mode in the anchor set batch. The logits obtained from the above;

[0158]

[0159]

[0160] That is, anchor set Sample data on;

[0161] This is a temperature hyperparameter used to regulate the output distribution characteristics of softmax, with a value between 0.1 and 0.5.

[0162] Final total loss:

[0163]

[0164] , All are single-modal loss weights, which are manually set according to specific training needs;

[0165] The I-only client model updates relevant modal expert parameters, shared base parameters, and optional task headers in reverse based on the loss.

[0166] The teacher signal used for alignment is the global teacher signal;

[0167] Loss calculation for T-only clients:

[0168] Similar to I-only, replace the representation with a text representation. and align with the same teacher prototype ,renew ;

[0169] Loss calculation for IT clients:

[0170] Calculate the task loss for each batch B. :

[0171]

[0172] Simultaneously optimize task loss and cross-modal consistency loss (e.g., contrastive learning / consistency constraints) on paired data:

[0173]

[0174] Final total loss:

[0175]

[0176] The full-modal loss weights are manually set according to specific training needs.

[0177] The IT client model updates relevant modal expert parameters, shared base parameters, and optional task headers in reverse based on the loss.

[0178] After this round of training, the class prototype and statistics (variance and class sample count) of each sample class c in the local dataset used for training are calculated:

[0179]

[0180]

[0181] In addition to the updated shared base module parameters and related modal expert parameters, the output also includes teacher package statistics. ;

[0182] S4 upload:

[0183] The client-side training in this round was completed locally and then uploaded to the server:

[0184] Shared dock update ;

[0185] Corresponding modal expert update ;

[0186] and contribution assessment scalars (such as );

[0187] The IT client additionally uploads statistics for each type of prototype. ;

[0188] The uploaded data consists of parameter increments, gradients, or statistics, not the original data.

[0189] S5 server-side teacher signal aggregation, such as Figure 4 As shown:

[0190] The server aggregates the class prototypes uploaded by IT clients to form a global teacher prototype:

[0191]

[0192] And maintain the variance / confidence estimate corresponding to the class prototype. :

[0193]

[0194] when Below the threshold This indicates that the number of samples for the corresponding class in the client's local dataset is small, the provided class prototype is not representative, and it is not included in the aggregation to improve reliability;

[0195] The server updates the global model on the server by aggregating the model updates uploaded by the clients participating in this round of training. The updated global model is then displayed in the anchor set. Obtain logits Used for teacher distillation, serving as a global teacher signal;

[0196] S6 server-side complementary contribution assessment and aggregation, such as Figure 5 As shown:

[0197] The server calculates the complementary contribution scores of the clients. Normalize the relevant indices within each mode group and then perform a linear combination:

[0198]

[0199] in:

[0200] This refers to alignment gain, such as the change in alignment loss on the anchor set before and after a client update, but it is not limited to a single alignment loss. It can include changes in alignment loss on the anchor set, cross-modal retrieval mAP gain, and improvement in consistency during contrastive learning.

[0201] For single-modal nodes, the decrease in KD loss on the reference anchor set:

[0202]

[0203] in For knowledge distillation loss;

[0204] For IT nodes, the reduction in cross-modal consistency loss on the reference anchor set:

[0205]

[0206] in This results in a loss of consistency.

[0207] To represent teacher quality, only IT has the capability, such as the negative logarithmic mean of the upload variance (but not limited to the negative logarithmic mean of the variance; other indicators include the variance truncation mean, confidence interval width, prototype stability, and inter-class separability indices).

[0208] This part is only available for IT nodes; for single-modal nodes, this part is 0. It is characterized by the negative logarithm of the discreteness of the prototype uploaded by the node; the smaller the discreteness, the higher the quality.

[0209]

[0210] Upload the set of categories of the prototype for node k;

[0211] Indicating stability (e.g., cosine similarity to the update direction of the group mean, but not limited to cosine similarity; gradient norm drift, direction consistency, and KL algorithm for the update distribution can also be used); consistency between the update direction and the group mean direction:

[0212]

[0213] This represents the average direction of the initial model update issued in this round.

[0214] The implementation in this embodiment takes into account and For ease of local computation, these two parts are calculated locally as scalars and then directly uploaded; subsequently, softmax is used within each modality group to obtain the group weights.

[0215]

[0216] And by introducing cross-group quotas (anchor protection), cross-group aggregate weights are obtained:

[0217]

[0218] First, competition within the group (softmax), then cross-group quotas ( This is to simultaneously address the issues of "intra-group quality disparity" and "inter-group IT dilution"; ordinary single-layer weighted averages cannot satisfy both simultaneously; servers aggregate shared base parameters and expert parameters respectively.

[0219] expert Intra-group aggregation:

[0220]

[0221] Shared dock Use cross-group weights polymerization:

[0222]

[0223] Once completed, proceed to the next round of training (t+1).

[0224] If the preset number of iterations has been completed or the training performance of the models on the local dataset for each client has converged, then the federated learning phase is considered complete.

[0225] After the federated learning phase is completed, each client uses the iterated model to adapt the new data to the same task. In this embodiment, the nodes that participate in federated learning can obtain a stronger global model than the local training model as a new medical diagnosis model. At the same time, IT modal knowledge is better protected by quotas, and the role of full-modal medical data can be better utilized when IT modal knowledge accounts for a very small proportion.

[0226] The innovative aspects of this example are as follows:

[0227] A parameter structure: the trainable adaptation parameters are split into shared base adaptation parameters θs and modality mask-triggered expert adaptation parameters θ. I / θ T / θ IT (and optional gating φ), and in federated training, only the corresponding expert parameters and θ are sent and returned according to the client-dominated modal mask. s This mechanism isolates update interference under different mode-deficient conditions.

[0228] One form of teacher signal: For IT clients with paired text and images, generate and upload category-level teacher statistical signals during local training (prototype). Variance / Confidence ,count Server aggregation forms a global teacher prototype. In addition, it provides a unified semantic anchor point for I-only / T-only clients by optionally using top-k logits and temperature T based on anchor set A to mitigate semantic drift.

[0229] An aggregation constraint: The server calculates complementary contribution scores. (by alignment gain) Teacher quality ,stability (Constructed by, etc.), first, softmax normalization is performed within each modality group to obtain the intra-group weights, and then cross-group quotas are applied. and constraints An anchor point protection strategy is implemented to prevent the signals from a few IT teachers from being diluted during aggregation, and to update the shared base parameters and individual expert parameters separately.

[0230] Teacher and communication compression implementation: Teacher logits can be compressed and transmitted according to top-k probabilities and indices; the client preferably freezes the basic model θ0 and only updates PEFT and a small number of statistics, thereby reducing communication costs and reducing the exposure of privacy risks.

Claims

1. A federated multimodal large model adaptation method based on modal complementarity and cooperation, characterized in that, The system comprises a server and multiple clients, and the method includes the following steps: S1 Initialization Preparation: The client first performs modal masking on the local dataset, modeling the modal availability of the data as a first-order variable, namely the modal availability mask m. m identifies images as I, text as T, and text and images as IT. The client determines the dominant mask based on the data it holds locally. That is, clients that only hold image data are I-only clients, clients that only hold text data are T-only clients, and clients that hold paired datasets of images and text are IT clients. The client's dominant mask is backed up on the server side. Based on the client's dominant mask, the server distributes the initial basic model that conforms to the client's characteristics to all clients participating in federated learning. The local dataset after being partitioned using modal masks is in the following form: , in and These represent paired image and text data, respectively. If a modality is missing, then... ; The label represents the sample in the dataset, that is, the semantic category to which the sample belongs; S2 grouping and distribution: The server selects the client set to participate in round t. The server sends data to the participating client set. Each client in the process sends out adaptation parameters; S3 client local training: The client freezes the main parameters of the base model. Under these conditions, only update the PEFT parameter in the adaptation parameters; The PEFT parameters are injected into the basic model in the form of residuals: for the output representation of any modal tower, let the frozen forward be... Then the equivalent forward representation after PEFT injection is: + + ， in This represents a lightweight residual branch implemented using PEFT parameters. Selection is triggered by the modal availability mask; Let the frozen visual tower be The frozen text tower is , respectively and Do Normalization ,get: Image representation: ， Text representation: ， Optional gating fusion can be added to the IT client: ， The representations obtained by each client are then passed forward for loss calculation; After this round of training, the IT client calculates the class prototype and statistics of each sample class c in the local dataset used for training as the teacher package: ， ， In addition to updated shared dock module parameters and related modality expert parameters, the IT client output also includes teacher package statistics. ; S4 upload: The client-side training in this round was completed locally and then uploaded to the server: Shared dock update ; Corresponding modal expert update ; Contribution assessment scalar; Additional Teacher Package Upload Statistics from IT Clients ; S5 server-side teacher signal aggregation: The server aggregates the class prototypes uploaded by IT clients to form a global teacher prototype: ， And maintain the variance statistics corresponding to the class prototype. : ， when Below the threshold This indicates that the number of samples for the corresponding class in the client's local dataset is small, the provided class prototype is not representative, and it is not included in the aggregation to improve reliability; The server updates the global model on the server by aggregating the model updates uploaded by the clients participating in this round of training. The updated global model is then displayed in the anchor set. Obtain logits Used for teacher distillation, serving as a global teacher signal; S6 Server-Side Complementary Contribution Assessment and Aggregation: The server calculates the complementary contribution scores of the clients. Normalize the relevant indices within each mode group and then perform a linear combination: ， in: Indicates alignment gain; Indicates teacher quality; Indicates stability; Then, softmax is used within each modality group to obtain the group weights: ， This is the reciprocal form of the weighted temperature hyperparameter; And cross-group quotas are introduced to obtain cross-group aggregate weights: ， The servers aggregate shared base parameters and expert parameters respectively: expert Intra-group aggregation: ， Shared dock Use cross-group weights polymerization: ， After completion, proceed to the next training round t+1; If the preset number of iterations has been completed or the training performance of the models on the local dataset for each client has converged, then the federated learning phase is considered complete. After the federated learning phase is completed, each client uses the iterated model to perform model adaptation for the same task on the new data.

2. The federated multimodal large model adaptation method based on modal complementarity and cooperation according to claim 1, characterized in that, The adaptation parameters in step S2 include the following: The server distributes shared docking parameters to all clients in this round. ; The server, based on the dominant mask backup of each client on the server, sends model parameters that match the characteristics of each client. If the client's dominant mask is I-only, then image expert parameters are also sent. If the client-side master mask is T-only, then additional text expert parameters will be sent. If the client-dominant mask is IT, then the IT expert parameters will be sent simultaneously with the image and text expert parameters. and optional gating parameters ; The server distributes the teacher package statistics for the current round and the global IT prototype aggregated from the previous round in each round. In public anchorage The logits soft tag on top.

3. The federated multimodal large model adaptation method based on modal complementarity and cooperation according to claim 1, characterized in that, The representation obtained in step S3 is further passed forward for loss calculation, for I-only clients: Calculate the task loss for each batch B. : ， Furthermore, class prototypes and statistics are introduced as teacher signals to calculate the alignment complementarity loss: ， Hyperparameters used to maintain numerical stability; in For image encoding representation, logits distillation can be optionally performed on the anchor set batch b. Indicates the corresponding mode in the anchor set batch The logits obtained from the above; That is, anchor set Sample data on; This is a temperature hyperparameter, with values ​​ranging from 0.1 to 0.

5. Final total loss: ， , All are single-modal loss weights, which are manually set according to specific training needs; The I-only client model updates relevant modal expert parameters, shared base parameters, and optional task headers in reverse based on the loss. The teacher signal used for alignment is the global teacher signal.

4. The federated multimodal large model adaptation method based on modal complementarity and cooperation according to claim 3, characterized in that, The representation obtained in step S3 is further passed forward for loss calculation, for T-only clients: Similar to the loss calculation process for I-only clients, the representation is replaced with a text representation. and align with the same teacher prototype ,renew .

5. The federated multimodal large model adaptation method based on modal complementarity and cooperation according to claim 3, characterized in that, The representation obtained in step S3 is further passed forward for loss calculation, for IT clients: Calculate the task loss for each batch B. : ， Simultaneously optimize task loss and cross-modal consistency loss on paired data: ， Final total loss: ， The full-modal loss weights are manually set according to specific training needs. The uniform temperature hyperparameter is set between 0.1 and 0.

5. The IT client model updates relevant modal expert parameters, shared base parameters, and optional task heads in reverse order based on the loss.

Images