Federal multi-modal learning adaptive method and system for dynamic modal availability

Abstract

This invention discloses an adaptive method and system for federated multimodal learning oriented towards dynamic modal availability, belonging to the field of federated learning technology. The method includes: recording and predicting the modal state of each client to obtain the probability of modal availability in the next round; calculating the switching intensity and stability coefficient to determine the subset of clients participating in the current training round; determining modal gating for the visual branch, text branch, and fusion branch based on the modal state, driving the local model to complete low-rank training and generate local low-rank parameters; determining whether the client has experienced long-term single-modal degradation or high-frequency switching based on the stability coefficient; if so, triggering distillation constraints, representation alignment, and prototype cache replay to obtain consolidated low-rank parameters and upload them to the server; if not, uploading the local low-rank parameters to the server; and the server performing global aggregation based on modal awareness. This invention has a clear necessity in improving the stability, efficiency, and deployment practicality of dynamic federated multimodal systems.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a federated multimodal learning adaptive method and system oriented towards dynamic modality availability. Background Technology

[0002] Federated Learning (FL) achieves distributed collaborative training under privacy constraints by retaining the original data locally at each participating end and exchanging only model parameters or gradients. With the widespread application of multi-source information such as vision, text, speech, and sensor signals in scenarios such as social media governance, disaster event analysis, and medical question answering, federated multimodal learning is gradually becoming an important direction for improving the model's perceptual capabilities and cross-scenario generalization ability. Compared to unimodal methods, multimodal models can simultaneously utilize image content, text semantics, and their interaction relationships, enabling more stable discrimination of complex semantics, contextual conflicts, and targets with weak cues. Therefore, it has significant value in tasks such as hate content recognition, crisis event classification, and medical image and text understanding.

[0003] However, modalities in real-world deployment environments are not always consistently stable and available. Clients may exhibit multiple states at different times, such as complete bimodal, single-image, single-text modal, or even short-term failures, due to factors like camera shutdown, image acquisition failure, missing text fields, limited network bandwidth, or switching acquisition strategies. This type of "time-varying modal availability" directly impacts the quality of local updates, global aggregation stability, and the continuity of learned cross-modal knowledge during federated training. Therefore, enabling federated multimodal models to perceive temporal changes in client modal states and adaptively adjust training and inference paths without leaking local data has become a critical technical problem that urgently needs to be solved in this field.

[0004] Currently, existing technologies typically develop in two directions. The first type of approach focuses on general frameworks for federated multimodal collaborative training. The basic idea is to build visual branches, text branches, or cross-modal fusion modules on each client side, and then achieve global aggregation through mechanisms such as federated averaging, federated optimization, or efficient parameter fine-tuning. Related research shows that multimodal information can improve the performance of federated models under privacy constraints, but this often presupposes that clients can stably provide a given combination of modalities over a long period, or at least maintain a relatively fixed modal configuration during training. The second type of approach addresses robust modeling methods for missing modalities. These typically maintain recognition performance in centralized or single-machine environments by using modal completion, shared representation learning, cue learning, or distillation transfer, even when some modalities are missing. For example, the missing modality multimodal cueing method proposed by Lee et al. at CVPR 2023 introduces lightweight cue parameters for different modality missing cases, enabling the model to maintain good visual recognition performance even when modalities are incomplete during training or testing. In 2024, Sun et al. proposed FedCola, which investigated the problem of transferable multimodal collaboration in federated scenarios when different clients hold data of different modalities. This improved performance on visual language tasks by narrowing the gap between intramodal and cross-modal data. In 2024, Yuan et al. proposed mmFedMC, further considering client selection and modality selection from a communication constraint perspective to reduce the upload overhead of multimodal federated training. These schemes have advanced related technologies from the perspectives of robustness to missing modalities, cross-client modality complementarity, and communication efficiency, providing an important foundation for federated multimodal learning.

[0005] Despite positive progress in existing solutions, significant shortcomings remain in practical federated deployments addressing dynamic modality availability. For example: 1) Most methods treat the client modality set as a static condition, defining which modalities a client possesses only at the start of training, lacking continuous tracking and prediction of modality availability evolution over time. When a client experiences periodic sensor failures, bandwidth fluctuations, or strategically shutting down a modality, the model cannot adjust its training path in advance, easily leading to local update oscillations and aggregation bias. 2) Existing federated multimodal frameworks typically employ fixed structures or fixed fusion orders. Even if a modality is unavailable in the current round, related branches are still retained or coarsely skipped. This leads to two types of problems: firstly, unavailable branches may generate invalid computation and parameter synchronization overhead; secondly, simply shutting down branches may destroy the original fusion representation, making it difficult for the model to quickly return to a stable state after modality recovery. 3) Many methods for missing modalities primarily focus on single or independent missing cases, lacking systematic handling of long-term continuous missing modalities, intermittent recovery, and knowledge forgetting issues in multi-round federated training. When a client consistently provides only single-modal data, the model tends to gradually favor the currently available modality, weakening previously learned cross-modal correspondences and ultimately affecting the global model's ability to model complex semantic relationships. 4) While some existing solutions consider modality selection or communication costs, they haven't yet incorporated "modal state change history" as a core decision-making basis, making it difficult to strike a balance between availability variations, computational budget, and global knowledge preservation. This leads to potential performance degradation, training instability, and low resource utilization in real-world scenarios such as social media analysis and disaster site perception when faced with frequent changes in client modality access conditions. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a federated multimodal learning adaptive method and system oriented towards dynamic modality availability. In this method and system, client modality configurations are not assumed to be constant; instead, modality availability is treated as an observable, accumulative, and predictable temporal state. First, the historical modality states of each client are recorded and modeled. Then, the visual branch, text branch, and their fusion path are dynamically routed based on the current availability, allowing the training and inference processes to flexibly adjust to environmental changes. Furthermore, to avoid cross-modal knowledge forgetting during long-term single-modality phases, this invention introduces a resilient knowledge consolidation mechanism. Through periodic distillation and representation alignment, previously learned complementary information is maintained, ensuring that existing knowledge can be continued even when a modality is temporarily missing, and that collaborative effects can be quickly reconstructed after the modality is restored. Compared to existing static configuration or one-time missing handling schemes, this invention simultaneously considers time-varying modality perception, structural-level dynamic adaptation, and knowledge continuity in federated training. Therefore, it is clearly necessary to improve the stability, efficiency, and deployment practicality of dynamic federated multimodal systems.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] Federated multimodal learning adaptive methods for dynamic modal availability include:

[0009] In the current round of the federation, a set of candidate clients is determined, and the server sends global parameters to the clients in the candidate client set; each client in the candidate client set collects and parses image streams and text streams to construct the current round's modal state; based on the client's modal state, client feasibility is predicted to obtain the probability of the next round's modality availability; the switching intensity and stability coefficient are calculated, and the subset of clients participating in this round of training is determined based on the client's probability of the next round's modality availability and stability coefficient;

[0010] Each client in the client subset determines the modality gating of the visual branch, text branch, and fusion branch based on the modality state, and selects a shared adapter or a private adapter based on the client's next round modality availability probability, driving the local model to complete low-rank training and generating local low-rank parameters and a unified task representation.

[0011] Based on the stability coefficient, it is determined whether the client has experienced long-term single-modal degradation or high-frequency switching. If so, distillation constraints, representation alignment, and prototype cache replay are triggered to update the current local model parameters, obtain consolidated low-rank parameters, and upload the consolidated low-rank parameters to the server. If not, the local low-rank parameters are uploaded to the server. The server obtains the updated global parameters based on the execution of modality-aware global aggregation and distributes the updated global parameters to each client participating in the training, thus entering the next federated round.

[0012] Preferably, the step of predicting client feasibility based on the client's modal state to obtain the probability of modal availability in the next round includes:

[0013] The availability of image and text modalities is represented in binary form to construct modal states. The formula is as follows:

[0014]

[0015] in, Indicates the client In the round The image modality can be indicated by a quantity. Indicators can be used to represent text modalities;

[0016] By combining the current modal state with the previous historical state to encode the modal evolution trend, an exponentially smoothed historical state is constructed. The formula is as follows:

[0017]

[0018] in, Indicates the client In the round Smooth historical availability representation, Indicates the historical retention factor;

[0019] Current modal state With historical status After concatenation, the input to the prediction function yields the client's next round modality availability probability. The formula is as follows:

[0020]

[0021] in, Indicates the client In the round The modalities can be predicted using probability. and These represent the prediction parameter matrix and the bias term, respectively. This represents the element-wise Sigmoid function.

[0022] Preferably, the calculation of switching intensity and stability coefficients, and the determination of the subset of clients participating in this round of training based on the client's next round modality availability probability and stability coefficients, includes:

[0023] The differences in modal states between adjacent rounds are statistically analyzed to generate switching intensity. ,as follows:

[0024]

[0025] in, Indicates the intensity of mode switching. Representing vectors Norm;

[0026] Determine the client's stability coefficient based on the handover intensity. The formula is as follows:

[0027]

[0028] in, and These represent the components of the image modality and the text modality in the historical state, respectively. Indicates switching the penalty coefficient;

[0029] Based on the client's stability coefficient and the probability of modality availability in the next round, an effectiveness score is calculated from the candidate client set. The K clients with the highest validity scores are selected as the subset of clients to participate in this round of training, as follows:

[0030]

[0031] in, This indicates that the highest score is selected. One client, This represents the predicted availability gain coefficient.

[0032] Preferably, the process of driving the local model to complete low-rank training and generate local low-rank parameters and a unified task representation includes:

[0033] Generate routing gating vectors based on modal states. And apply it to the visual branch, text branch, and fusion branch, as follows:

[0034]

[0035] in, Indicates the visual branch gate value. Indicates the text branch gate value. Indicates the fusion branch gate value;

[0036] Construct image features respectively Text features and fusion features ,as follows:

[0037]

[0038] in, Represents the cross-modal fusion operator;

[0039] Considering that some clients are in modality-deficient, fusion-modal, or unimodal scenarios, a unified task representation is constructed. ,as follows:

[0040]

[0041] in, Indicates the client In the round A unified task representation;

[0042] Local model training is performed based on a unified task representation, and the task loss during the training process is considered. The formula is as follows:

[0043]

[0044] in, Indicates the client In the round The number of local samples, Indicates the first The supervision target for each sample This indicates the currently trainable parameters for this client;

[0045] To suppress excessive drift caused by local data offset under dynamic modal conditions, a local loss is constructed using proximal constraints on the global model for local updates. This local loss is:

[0046]

[0047] in, Represents the proximal regularity coefficient. This represents the set of parameters that are active and allowed to be updated in the current round. Indicates the first A global parameter.

[0048] Preferably, the steps of triggering distillation constraints, representation alignment, and prototype cache replay to update the current local model parameters and obtain consolidated low-rank parameters include:

[0049] Teacher output in the complete historical modality Student output in the currently available modalities Distillation constraints are applied to determine distillation losses. ,as follows:

[0050]

[0051] in, Indicates the distillation temperature. Indicates the Kullback-Leibler divergence;

[0052] Establish a representation alignment loss between the fused representation and the single-modal representation. ,as follows:

[0053]

[0054] in, Indicates the client The fused representation prototype cached in the most recent full modal state;

[0055] The local loss, distillation loss, and representation alignment loss are combined into a knowledge consolidation objective. ,as follows:

[0056]

[0057] in, Indicates the distillation intensity coefficient. This represents the alignment strength coefficient.

[0058] Preferably, the modality-aware global aggregation includes:

[0059] The base weights are calculated based on the number of client samples, and then the stability coefficient is added to obtain the aggregate weights. The formula is as follows:

[0060]

[0061] in, Indicates the client In the round Local sample size Indicates the client In the round stability coefficient, This represents a lower bound constant to prevent the denominator from being zero;

[0062] The server updates global parameters as follows:

[0063]

[0064] in, Indicates the client Uploaded partial parameters, This indicates the global parameters for the next round.

[0065] Preferably, it further includes: when using the FedYogi, FedAdam, FedAdagrad, or FedAvgM adaptive aggregation strategy, writing the client incrementally as Then calculate the modal sensing mean increment. The formula is as follows:

[0066]

[0067] in, Indicates the global parameters for the current round. This represents the weighted average increment;

[0068] When using adaptive server optimization, the server updates global parameters as follows:

[0069]

[0070] in, Indicates the server learning rate. Indicates by The obtained first-order momentum term, Represents a second-order statistical term. This represents the numerical stability constant.

[0071] Based on the above, this invention also discloses a federated multimodal learning adaptive system for dynamic modality availability, comprising:

[0072] The temporal modal availability modeling module records the modal state of each client, predicts client feasibility based on the client's modal state, obtains the probability of modal availability in the next round, calculates the switching strength and stability coefficient, and determines the subset of clients participating in this round of training based on the client's probability of modal availability in the next round and the stability coefficient.

[0073] The dynamic architecture routing and local training module is used to determine the modality gating of the visual branch, text branch and fusion branch according to the modality state of each client in the client subset, select the shared adapter or private adapter according to the next round modality availability probability of the client, drive the local model to complete low-rank training, and generate local low-rank parameters and unified task representation.

[0074] The elastic knowledge consolidation and federated aggregation module is used to determine whether a client has experienced long-term single-modal degradation or high-frequency switching based on the stability coefficient. If so, it triggers distillation constraints, representation alignment, and prototype cache replay, updates the current local model parameters, obtains consolidated low-rank parameters, and uploads the consolidated low-rank parameters to the server. If not, it uploads the local low-rank parameters to the server. The server obtains the updated global parameters based on the modality-aware global aggregation and distributes the updated global parameters to each client participating in the training, thus entering the next federated round.

[0075] Based on the above technical solution, the beneficial effects of the present invention are:

[0076] 1) Significantly improves recognition accuracy under periodic mode absence conditions: This invention uses temporal modality availability modeling to predict the next available modality for the client and pre-adjusts the training path. As shown in Table 1, the accuracy of this invention reaches 72.6%, which is higher than FedAvg's 69.8% and FedAdam's 71.2%, effectively reducing the classification performance fluctuations caused by periodic failures.

[0077] 2) Improved global discrimination stability under random interruption environments: This invention introduces modality-aware aggregation on the server side, jointly encoding the client sample size and stability coefficient to avoid excessive perturbation of global parameters by clients with transient modality loss. As shown in Table 2, the macro-average F1 of this invention reaches 77.8%, which is 2.2 percentage points higher than FedProx's 75.6%.

[0078] 3) Enhanced knowledge retention after long-term single-modal training: This invention utilizes an elastic knowledge consolidation module to perform periodic distillation and representation alignment of historical complete modal states, preventing the client from forgetting fusion semantics after long-term single-modal training. As shown in Table 3, the F1 score of this invention reaches 63.0%, higher than FedAvg's 59.8%, indicating that cross-modal knowledge is stably retained.

[0079] 4) Shortened readaptation time after modality recovery: This invention retains the necessary low-rank context and fusion interface before the modality becomes available again, allowing reconnection to the federated process without significant retraining of the recovery branch. As shown in Table 3, the stable recovery rounds of this invention are 7, fewer than the 9 rounds of FedYogi and FedAdam, demonstrating faster recovery efficiency.

[0080] 5) Reduce unnecessary computation and communication consumption in dynamic scenarios: This invention selectively activates visual, text, and fusion branches according to the currently available modalities and prediction results, synchronizing only the necessary low-rank adaptation parameters. As shown in Table 4, this scheme binds "modal state awareness" and "parameter on-demand update" together, which can reduce idle training and redundant uplink communication in modal-deficient states.

[0081] 6) Improved convergence stability in multi-scenario hybrid federated training: This invention moderately reduces the update weights of clients that frequently switch, while retaining the contributions of stable clients, preventing the global model from being swayed by local abnormal states. As shown in Tables 1 and 2, this invention maintains a leading position in different dynamic scenarios, indicating that its convergence path has better continuity and generalization consistency.

[0082] 7) Improve the adaptability of the solution to different business tasks: This invention does not rely on a single data type or a fixed missing-mode pattern, but covers identification, perception, and question-answering tasks through unified state coding, routing control, and aggregation constraints. As shown in Tables 1 to 3, this method can be applied to multiple types of federated multimodal businesses, such as content moderation, emergency perception, and medical auxiliary analysis.

[0083] 8) Forming a closed-loop control system that can be engineered and implemented: This invention organizes "availability prediction, dynamic routing, knowledge consolidation, and aggregation correction" into a consistent federated control link. As shown in Table 4, each module corresponds to a clear control object and engineering benefits, thereby improving performance indicators and enhancing the system's continuous operation capability under real network fluctuations and equipment failures. Attached Figure Description

[0084] Figure 1 This is a schematic diagram of a federated multimodal learning adaptive method for dynamic modality availability in one embodiment;

[0085] Figure 2This is a flowchart of a federated multimodal learning adaptive method for dynamic modal availability in one embodiment;

[0086] Figure 3 This is a flowchart illustrating a scenario in which a federated multimodal learning adaptive method for dynamic modality availability is applied to a periodically missing modality federated training of a cross-departmental hate content moderation network.

[0087] Figure 4 This is a flowchart illustrating a scenario where a federated multimodal learning adaptive method for dynamic modal availability is applied to a disaster emergency terminal network for rapid recovery from random outages. Detailed Implementation

[0088] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0089] like Figure 1 , 2As shown, this embodiment provides a federated multimodal learning adaptive method for dynamic modal availability, hereinafter referred to as TMA. This method addresses real-world environments where client-side image modalities, text modalities, and their fused representations are alternately available, intermittently missing, or chronically degraded across different federation rounds. It follows a closed-loop technical approach of "first perception, then routing, and finally consolidation." Specifically, it first constructs a temporal modal availability modeling module to encode the modal state, historical trends, and predicted reachability of each client in the current round, obtaining an availability representation that can be used for subsequent training control. Then, it constructs a dynamic architecture routing and local training module to perform fine-grained activation and freezing between visual branches, text branches, fusion branches, and low-rank adaptation parameters based on the current client's modal state, reducing invalid computation and maintaining consistency between local updates and input conditions. Finally, it constructs a resilient knowledge consolidation and federated aggregation module to maintain cross-modal knowledge continuity through periodic distillation, representation alignment, and modality-aware global aggregation after frequent modal switching or long-term single-modal training, preventing the global model from being dominated by short-term missing modes. This method can be deployed on various federated multimodal tasks such as hate content recognition, disaster event awareness, and medical image-text question answering. It is compatible with aligned modalities, missing modalities, mixed modalities, and unimodal scenarios, as well as federated optimization strategies such as FedAvg (Federated Averaging), FedProx (Federated Proximal), FedYogi (Federated Yogi), FedAdam (Federated Adam), FedAdagrad (Federated Adagrad), and FedAvgM (Federated Averaging Momentum). It can also work in conjunction with the LoRA (Low-Rank Adaptation) framework for efficient parameter fine-tuning. The method includes:

[0090] Step 1: In the current federated round, a candidate client set is determined, and the server sends global parameters to the clients in the candidate client set; each client in the candidate client set collects and parses image streams and text streams to construct the current round modal state; based on the client's modal state, client feasibility prediction is performed to obtain the probability of modality availability in the next round; the switching intensity and stability coefficient are calculated, and the subset of clients participating in this round of training is determined according to the client's probability of modality availability in the next round and the stability coefficient.

[0091] In this embodiment, the actual image and text modalities available to each client in each round are recorded to form a historical state cache. Furthermore, short-term switching intensity, long-term availability, and the next round's availability prediction results are calculated. This module transforms the originally discrete and passive modality loss phenomenon into quantifiable control signals, providing a basis for subsequent dynamic branch activation, training scheduling, and aggregate weight correction.

[0092] 1) Client-side modal state coding

[0093] In each federal round This invention is for the client. Constructing modal states The image and text modalities are represented binaryally to indicate their availability. When the client has both image and text input capabilities, the state vector is bimodal; when the client is in a single-modal or missing-modal state, the position corresponding to the missing modality is explicitly set to zero. This modality state vector not only serves the routing decision in the current round but is also written to the historical queue as the basis for time-varying modality statistics.

[0094]

[0095] in, Indicates the client In the round The image modality can be indicated by a quantity. Indicators can be used to represent text modalities.

[0096] To avoid the controller making unstable judgments based solely on a single round of observations, this invention further constructs an exponentially smoothed historical state. The current modal state and the previous historical state are used to jointly encode the modal evolution trend. When the modal state experiences periodic failure or gradual recovery, the historical state can provide a more stable control basis than the original binary label.

[0097]

[0098] in, Indicates the client In the round Exponential smoothing of historical states, This represents the historical retention factor.

[0099] Furthermore, to reflect the intensity of mode switching, this invention statistically analyzes the state differences between adjacent rounds and generates switching intensity. When a client frequently switches from bimodal to unimodal or reverts from unimodal to bimodal across multiple rounds, its score increases, allowing subsequent modules to reduce the weight of aggressive updates.

[0100]

[0101] in, Indicates the intensity of mode switching. Representing vectors Norm.

[0102] 2) Availability prediction and round-robin scheduling

[0103] After obtaining historical states, this invention further predicts modality availability for the next round or several future rounds, in order to adjust training paths and client participation priorities in advance. For the client... The current modal state With historical status After concatenation, the input to the prediction function yields the available probability vector for the next round of modes. When the recovery probability of a certain mode is high, the system can pre-reserve the low-rank parameters of the corresponding branch; when the recovery probability is low, the robust representation of the single mode can be strengthened first.

[0104]

[0105] in, Indicates the client In the round The modalities can be predicted using probability. and These represent the prediction parameter matrix and the bias term, respectively. This represents the element-wise Sigmoid function.

[0106] To apply the prediction results to federated round-robin scheduling, this invention defines a client stability coefficient. Taking into account both historical average availability and switching intensity, higher scheduling priority is given to clients with higher predictability, while smoothing constraints are imposed on clients with frequent fluctuations. This coefficient can be directly applied to sampling sorting or aggregate weight correction.

[0107]

[0108] in, and These represent the components of the image modality and the text modality in the historical state, respectively. This indicates switching the penalty coefficient.

[0109] Furthermore, the present invention addresses the rounds. Candidate client set Perform modal sensing sampling to select the participating set. The number of selected clients is consistent with the system configuration of sample_clients, but the sorting criteria no longer rely solely on random sampling. Instead, availability predictions and historical stability are combined to improve the effectiveness of each update on the global model.

[0110]

[0111] in, This indicates that the highest score is selected. One client, This represents the predicted availability gain coefficient.

[0112] Step 2: Each client in the client subset determines the modality gating of the visual branch, text branch, and fusion branch based on the modality state. Based on the client's next round modality availability probability, a shared adapter or a private adapter is selected to drive the local model to complete low-rank training and generate local low-rank parameters and a unified task representation.

[0113] In this embodiment, it is not assumed that the visual branch, text branch, and fusion branch always participate synchronously. Instead, before each round of local training, different branches, adapters, and loss terms are gating based on the output modality state and predicted probability. For bimodal clients, a complete visual-language collaborative link can be enabled; for unimodal clients, only necessary branches are retained and compatibility with the global model is maintained through a shared representation interface; for clients expected to recover another modality, lightweight parameter context is retained to reduce the cost of readaptation after recovery.

[0114] 1) Modal gating and branch activation

[0115] For the client In the round The invention first generates a routing gating vector based on the modal state vector as input. This is then applied to the visual encoder, text encoder, and fusion operator. If the image modality is missing, the visual branch is closed or enters a keep-alive state; if the text modality is missing, the text branch is closed; if both modalities are available simultaneously, the fusion path is activated. This mechanism ensures that the local forward propagation strictly matches the actual input conditions.

[0116]

[0117] in, Indicates the image branch gate value. Indicates the text branch gate value. This represents the fusion branch gating value.

[0118] After the gating vector is generated, the present invention constructs image features respectively. Text features and fusion features When a branch is closed, its output automatically degenerates into a zero vector or a buffered representation from the previous round, thus preventing undefined input from directly entering subsequent layers.

[0119]

[0120] in, This represents the cross-modal fusion operator. This represents the combined representation after fusion.

[0121] Considering that some clients are in modality-deficient, fusion-modality, or unimodal scenarios, this invention further constructs a unified task representation. :

[0122]

[0123] in, This represents the unified task representation for the client in the current round, which is used for subsequent classification, generation, or question-answering loss calculation.

[0124] 2) Low-rank adaptation-driven local training

[0125] During the model parameter update phase, this invention employs the LoRA low-rank adaptation framework for federated fine-tuning of large multimodal models, updating only the low-rank parameters related to the self-attention projection layer and necessary storable modules to reduce communication overhead and GPU memory burden. For any adapted weight matrix... The effective parameters on the client side are written as the sum of the base parameters and the low-rank increment, thereby absorbing local knowledge under the current modal conditions while preserving the stability of the main body of the large model, and updating the weights. ,as follows:

[0126]

[0127] in, Indicates the base weight of the freeze. and Indicates the client In the round A low-rank matrix, Indicates the low rank value. This represents the scaling factor.

[0128] To enable local training to be compatible with dynamic modality gating, this invention defines a modality-aware local task loss. Supervision signals are accumulated only on the currently available modalities and activation branches. For classification tasks, cross-entropy can be used; for question answering tasks, autoregressive generative loss can be used; both are represented by a unified method. drive.

[0129]

[0130] in, Indicates the client In the round The number of local samples, Indicates the first The supervision target for each sample This indicates the currently trainable parameters for this client.

[0131] When FedProx is selected as the federated algorithm, this invention introduces proximal constraints on the global model to suppress excessive drift caused by local data offsets under dynamic modal conditions. Its local loss is:

[0132]

[0133] in, Represents the proximal regularity coefficient. This represents the set of parameters that are active and allowed to be updated in the current round. Indicates the first A global parameter.

[0134] Step 3: Based on the stability coefficient, determine whether the client has experienced long-term single-modal degradation or high-frequency switching. If so, trigger distillation constraints, representation alignment, and prototype cache replay, update the current local model parameters, obtain the consolidated low-rank parameters, and upload the consolidated low-rank parameters to the server. If not, upload the local low-rank parameters to the server. The server obtains the updated global parameters based on the execution of modality-aware global aggregation and distributes the updated global parameters to each client participating in the training, entering the next federated round.

[0135] In this embodiment, instead of immediately aggregating all client updates after local training, knowledge consolidation is first performed on clients undergoing modality switching. Then, updates are differentially integrated based on sample size, modality stability, and aggregation algorithm type. This approach simultaneously preserves bimodal knowledge, limits noise propagation during aberrant switching, and maintains compatibility with existing federated optimizers.

[0136] 1) Periodic distillation and representation alignment

[0137] For clients that previously possessed bimodal capabilities but currently only offer unimodal input, this invention triggers a knowledge consolidation process at specified intervals. Specifically, it involves processing the teacher's output from historical full-modal states. Student output in the currently available modalities Distillation constraints are applied to ensure that the model retains as much prior understanding of missing modes as possible, even when only a portion of the input is observed at present.

[0138]

[0139] in, Indicates the distillation temperature. This represents the Kullback-Leibler divergence.

[0140] In addition to output distribution constraints, this invention also establishes an alignment loss between the fused representation and the single-modal representation, forcing the representation after long-term absence to remain close to the historical joint semantic center, so as to quickly reconstruct collaborative reasoning ability after modality recovery.

[0141]

[0142] in, Indicates the client The fused representation prototype cached in the most recent full modal state.

[0143] Ultimately, this invention combines local task loss, distillation loss, and representation alignment loss into a knowledge consolidation objective. When the client is in a stable bimodal state, the coefficients of the latter two items can be reduced, while their weight can be increased when the client is missing for a long time or switches frequently.

[0144]

[0145] in, Indicates the distillation intensity coefficient. This represents the alignment strength coefficient.

[0146] 2) Modality-aware global aggregation

[0147] On the server side, this invention performs modality-aware global aggregation on the low-rank parameter updates uploaded by each participating client. For the standard FedAvg format, the basic weights are first calculated based on the number of client samples, and then the stability coefficients from the previous module are added to obtain the aggregated weights. This prevents clients in the midst of severe modal oscillations from exerting excessive influence on the global model.

[0148]

[0149] in, Indicates the client In the round Local sample size This represents a lower bound constant to prevent the denominator from being zero.

[0150] Based on this, the server updates the global parameters:

[0151]

[0152] in, Indicates the client Uploaded partial parameters, This indicates the global parameters for the next round.

[0153] When using adaptive aggregation strategies such as FedYogi, FedAdam, FedAdagrad, or FedAvgM, this invention further writes the client incrementally as... Then calculate the modal sensing mean increment. In order to update in conjunction with server-side momentum terms or second-order statistics:

[0154]

[0155] in, Indicates the global parameters for the current round. This represents the weighted average increment.

[0156] For example, when using adaptive server optimization, the server update global parameters can be written as:

[0157]

[0158] in, Indicates the server learning rate. Indicates by The obtained first-order momentum term, Represents a second-order statistical term. This represents the numerical stability constant. Through this method, TMA can enable the server-side to update the temporal variation characteristics of explicit response modal availability while maintaining compatibility with existing federated training frameworks.

[0159] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0160] Experimental Data and Performance Verification

[0161] This study summarizes and verifies the technical effectiveness of TMA (Time-Series Modal Availability Model) under three key mechanisms: temporal modal availability modeling, dynamic architecture routing, and resilient knowledge consolidation, under dynamic modality loss, multi-scenario switching, and long-term single-modal operation. Verification tasks cover typical federated multimodal services such as hate content identification, disaster event awareness, and medical image-text question answering. Evaluation metrics include core indicators such as accuracy, macro-F1, area under the curve (AUC), and stable recovery rounds.

[0162] Table 1 Performance comparison in the scenario of periodic mode absence

[0163]

[0164] Table 1 shows that TMA can still maintain higher classification stability under periodic mode failure conditions. Its accuracy is 1.4 percentage points higher than FedAdam, indicating that its time-series prediction and branch pre-activation mechanism can buffer the performance fluctuations caused by mode switching in advance.

[0165] Table 2 Comparison of disaster event recognition performance in random modal interruption scenarios

[0166]

[0167] Table 2 shows that when visual or textual interruptions occur randomly on the client side, TMA suppresses unstable update propagation through modality-aware aggregation, improving Macro-F1 by 3.0 percentage points compared to FedAvg, while maintaining higher global discriminative consistency.

[0168] Table 3. Performance Comparison of Medical Question Answering in Long-Term Single-Modal Degradation Scenarios

[0169]

[0170] Table 3 shows that when some clients experience a long period of image modality deactivation, TMA uses elastic knowledge to consolidate and maintain cross-modal semantic memory, reducing the reconstruction rounds after recovery to 7 rounds, while continuing to lead in Accuracy and F1 metrics.

[0171] Table 4 Key Configurations and Engineering Benefit Mapping of TMA

[0172]

[0173] Table 4 shows that TMA is not a single improvement, but a closed loop composed of four types of control methods: prediction, routing, consolidation and aggregation. Therefore, it can simultaneously cover the three types of patent value: performance, recovery speed and deployment efficiency.

[0174] In one embodiment, a federated multimodal learning adaptive method for dynamic modality availability is also provided for application in the periodic mode-deficient federated training scenario of a cross-departmental hate content moderation network. See [link to relevant documentation]. Figure 3 The specific process includes the following:

[0175] This embodiment is deployed in a cross-institutional federated multimodal recognition network involving multiple content moderation agencies. Each agency's client continuously receives image and text complaint samples; however, due to client maintenance windows, transmission congestion, and privacy policy switching, image modalities are intermittently missing according to a periodic pattern. The system aims to maintain the federated model's stable ability to identify hate content without centralizing the original data, and to quickly return to a bimodal collaborative state after modality recovery, while controlling uplink communication and ineffective training costs.

[0176] M1: The server initializes the federated training task and distributes global parameters. The server starts 50 rounds of training tasks according to the federated configuration, maintains a candidate set of 10 clients, and samples 2 clients to participate in the update in each round; at the same time, it distributes local training constraints to each client, including shared LoRA parameters, base learning rate 2e-5, batch size 1, and gradient accumulation 16, and outputs a snapshot of the federated parameters that can be executed in the first round.

[0177] M2: The client collects the current modality availability status and writes it into the time series history. Each review node encodes the availability flag for the image stream and text stream respectively to form the current round modality state vector, and then performs exponential smoothing with the historical rounds to output client-level availability, switching intensity and time series features to be predicted, providing input for subsequent dynamic routing.

[0178] M3: The server-side tracker generates the next round of modal recovery predictions and sorts the sampling priorities. The controller calculates the next round of scheduling scores based on the client's historical stability and predicted availability probability, prioritizing the retention of nodes expected to recover bimodality for training; combined with the benefits of reducing invalid training rounds corresponding to "temporal modal availability modeling" in Table 4, a more effective subset of participating clients can be output.

[0179] M4: The client performs branch gating and parameter loading based on the prediction results. If only the text modality is available in the current round, the visual branch is frozen, the fusion interface is kept alive, and the shared LoRA adapter is loaded; if the dual modality is restored, the visual branch, text branch, and fusion branch are activated simultaneously. This step directly maps the modality state to the training path, avoiding idle computation triggered by missing modalities, and outputting a task representation consistent with the current input.

[0180] M5: The selected client performs local low-rank training and generates local updates. The client performs forward and backward propagation using the current task representation, updating only the low-rank parameters related to the self-attention projection layer. When there is significant data skew at a node, proximal constraints can be superimposed to stabilize the local update, resulting in local parameter increments and round-based losses. Combined with the periodic model loss results in Table 1, it can be verified that this training loop maintains recognition stability in dynamic scenarios.

[0181] M6: The system performs knowledge consolidation on long-term single-modal nodes. For nodes that lack image modalities for multiple consecutive rounds, the controller calls the historical complete modal output as the teacher signal, applies distillation constraints and representation alignment to the current student output, and avoids the global model forgetting visual semantics; combined with the stable recovery round data in Table 3, a reusable joint representation prototype can be reserved for subsequent modal recovery.

[0182] M7: The server performs modality-aware aggregation and writes back the new round of global parameters. The server calculates the aggregation weights based on the client sample size, stability, and local update quality, suppresses the contribution of nodes with drastic switching, and then completes the synthesis of global parameters. According to Tables 1 and 4, the aggregated model can improve accuracy and reduce redundant communication overhead during the model-deficient period, and output the next round of broadcast parameters.

[0183] M8: The server feeds back the updated model and scheduling strategy to the client, entering the next round of closed loop. New global parameters, sampling priorities, and stability constraints are sent back to each review node. Clients that have recovered bimodality reconnect to the complete training link, while clients that are still in a mode-missing state continue to retain the necessary context, thus forming a continuous operating closed loop of "prediction-routing-training-aggregation-feedback".

[0184] Implementation Results: In this scenario of periodic missing model review, TMA maintains stable recognition performance without uploading original image and text data. According to Table 1, the system accuracy reaches 72.6%, an improvement of 2.8 percentage points compared to FedAvg's 69.8%, and the Macro-F1 reaches 70.1%. Combined with the on-demand activation and freeze mechanism shown in Table 4, it can reduce invalid branch training and redundant parameter uploading in missing model rounds, balancing recognition accuracy and communication efficiency.

[0185] In one embodiment, a federated multimodal learning adaptive method for dynamic modal availability is also provided for rapid recovery from random outages in disaster emergency terminal networks. See [link to relevant documentation]. Figure 4 The specific process includes the following:

[0186] This embodiment is deployed in a federated sensing network composed of disaster emergency terminals from multiple regions. Each client consists of a mobile camera device, a text reporting module, and edge computing nodes. Under conditions of strong winds, power outages, or link jitter, the image modality and text modality may randomly lose connection, and newly connected clients often experience cold starts and localized domain bias. The system aims to achieve rapid recovery and personalized adaptation under random interruption conditions, enabling the server to continuously aggregate stable updates and preventing new clients from amplifying noise into the global model.

[0187] M1: The server establishes an emergency federation task and registers client capability profiles. The server registers the currently available modalities, expected online duration, and recent stability for both new and old clients. It still starts the task according to the federation rhythm of 50 rounds, 10 client candidates, and 2 sampled clients per round, and outputs control instructions containing global parameters and recovery decision thresholds.

[0188] M2: Upon access, the client reports image link, text link, and interruption event logs. Each client encodes the camera's online status, text alarm availability status, and recent link anomaly information into a modal history sequence, generates an executable input description for the current round, and outputs it to the time-series tracker as a basis for recovery prediction.

[0189] M3: The tracker calculates stability based on the history of random interruptions and identifies clients that recover first. For clients that frequently lose connection, the system reduces their immediate aggregation weight; for clients that recover quickly after a short period of jitter, their participation is retained first. Combined with the performance improvement in the random interruption scenario shown in Table 2, it can be confirmed that this step can suppress unstable clients from spreading errors to the global model.

[0190] M4: Clients switch between shared and private adapters based on the current modality. Stable online clients enable the shared LoRA adapter to participate in cross-node knowledge sharing; newly connected or frequently interrupted clients additionally load the private adapter to cache personalized compensation features, and determine the activation combination of visual, text, and fusion branches according to the available modalities, outputting a local model structure that matches the client's state.

[0191] M5: The client completes local training and performs selective updates on randomly lost samples. The client completes the current round of training with a batch size of 1 and gradient accumulation of 16, synchronizing only the necessary low-rank increments; for samples that cannot form complete bimodal supervision during the interruption, the loss is accumulated only on the available branches to avoid treating abnormal missing samples as normal joint inputs. Combined with the dynamic routing benefits in Table 4, the computational power consumption of the edge client can be reduced.

[0192] M6: The system performs resilient knowledge consolidation and recovery alignment on cold-start clients. Newly connected clients or clients that have just completed modality recovery receive teacher output from the most recent complete modality prototype cached by the server, and constrain the current training results with distillation loss and representation alignment loss; combined with the evidence in Table 3 that the number of stable recovery rounds is shortened to 7 rounds, the client can return to an aggregable state more quickly.

[0193] M7: The server performs selective aggregation based on stability, sample size, and algorithm strategy. The server performs modality-aware weighting on uploaded increments, prioritizing updates from stable clients and successfully recovered clients, while retaining only a limited contribution from high-frequency jitter nodes. According to Table 2, the system accuracy reaches 79.3%, and the Macro-F1 reaches 77.8%, demonstrating stronger global discriminative consistency under random interruption environments.

[0194] M8: The aggregation results are written back to each client and trigger the next round of personalized reconfiguration. The server sends back new shared parameters, private adapter retention suggestions, and subsequent sampling sorting to all clients; stable clients continue to participate in global sharing, and newly recovered clients are gradually incorporated into bimodal federated training according to their personalized paths, thus forming a closed-loop process that balances rapid recovery and long-term stability.

[0195] Implementation Results: In disaster emergency client networks where random outages and cold starts coexist, TMA (Transmission and Analysis) demonstrates its strengths in recovery speed and personalized benefits. According to Table 2, the system's accuracy reached 79.3%, an improvement of 2.8 percentage points compared to FedAvg's 76.5%, and its AUC reached 83.1%. Furthermore, Table 3 shows that the number of stable rounds after modal recovery was shortened to 7, significantly faster than FedAdam's 9 rounds. This indicates that the solution can simultaneously improve the stability of discrimination under random disconnection conditions and the recovery efficiency after new client access.

[0196] Based on the same inventive concept, this application also provides a system for implementing the above-described federated multimodal learning adaptive method for dynamic modal availability. The solution provided by this system is similar to the implementation described in the above method, and therefore will not be repeated here.

[0197] In one embodiment, a federated multimodal learning adaptive system for dynamic modality availability is also provided, comprising:

[0198] The temporal modal availability modeling module records the modal state of each client, predicts client feasibility based on the client's modal state, obtains the probability of modal availability in the next round, calculates the switching strength and stability coefficient, and determines the subset of clients participating in this round of training based on the client's probability of modal availability in the next round and the stability coefficient.

[0199] The dynamic architecture routing and local training module is used to determine the modality gating of the visual branch, text branch and fusion branch according to the modality state of each client in the client subset, select the shared adapter or private adapter according to the next round modality availability probability of the client, drive the local model to complete low-rank training, and generate local low-rank parameters and unified task representation.

[0200] The elastic knowledge consolidation and federated aggregation module is used to determine whether a client has experienced long-term single-modal degradation or high-frequency switching based on the stability coefficient. If so, it triggers distillation constraints, representation alignment, and prototype cache replay, updates the current local model parameters, obtains consolidated low-rank parameters, and uploads the consolidated low-rank parameters to the server. If not, it uploads the local low-rank parameters to the server. The server obtains the updated global parameters based on the modality-aware global aggregation and distributes the updated global parameters to each client participating in the training, thus entering the next federated round.

[0201] In the above embodiments, the modules of the federated multimodal learning adaptive system for dynamic modal availability can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0202] The embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0203] The above are merely preferred embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

1. A federated multi-modal learning adaptation method oriented to dynamic modal availability, characterized in that, include: In the current round of the federation, a set of candidate clients is determined, and the server sends global parameters to the clients in the candidate client set; Each client in the candidate client set collects and parses image streams and text streams to construct the current round modal state; based on the client's modal state, client feasibility is predicted to obtain the probability of the next round's modality availability; Calculate the switching strength and stability coefficients, and determine the subset of clients participating in this round of training based on the client's next round modality availability probability and stability coefficient; Each client in the client subset determines the modality gating of the visual branch, text branch, and fusion branch based on the modality state, and selects a shared adapter or a private adapter based on the client's next round modality availability probability, driving the local model to complete low-rank training and generating local low-rank parameters and a unified task representation. Based on the stability coefficient, it is determined whether the client has experienced long-term single-modal degradation or high-frequency switching. If so, distillation constraints, representation alignment, and prototype cache replay are triggered to update the current local model parameters, obtain consolidated low-rank parameters, and upload the consolidated low-rank parameters to the server. If not, the local low-rank parameters are uploaded to the server. The server obtains the updated global parameters based on the execution of modality-aware global aggregation and distributes the updated global parameters to each client participating in the training, thus entering the next federated round.

2. The federated multi-modal learning adaptation method for dynamic modality availability according to claim 1, wherein, The client-based modal state-based prediction of client feasibility to obtain the probability of modal availability in the next round includes: The image modality and the text modality are respectively represented in a binary manner whether available or not, and the modality state is constructed , as follows: wherein, representing a client in a round of image modalities available indication, representing a text modalities available indication; By combining the current modal state with the previous historical state to encode the modal evolution trend, an exponentially smoothed historical state is constructed. The formula is as follows: in, Indicates client In the round Smooth historical availability representation, Indicates the historical retention factor; Current modal state With historical status After concatenation, the input to the prediction function yields the client's next round modality availability probability. The formula is as follows: in, Indicates client In the round The modalities can be predicted using probability. and These represent the prediction parameter matrix and the bias term, respectively. This represents the element-wise Sigmoid function.

3. The federated multimodal learning adaptive method for dynamic modal availability according to claim 2, characterized in that, The calculation of switching strength and stability coefficients, and the determination of a subset of clients participating in this round of training based on the client's next round modality availability probability and stability coefficient, include: The differences in modal states between adjacent rounds are statistically analyzed to generate switching intensity. ,as follows: in, Indicates the intensity of mode switching. Representing vectors Norm; Determine the client's stability coefficient based on the handover intensity. The formula is as follows: in, and These represent the components of the image modality and the text modality in the historical state, respectively. Indicates switching the penalty coefficient; Based on the client's stability coefficient and the probability of modality availability in the next round, an effectiveness score is calculated from the candidate client set. The K clients with the highest validity scores are selected as the subset of clients to participate in this round of training, as follows: in, This indicates that the highest score is selected. One client, This represents the predicted availability gain coefficient.

4. The federated multimodal learning adaptive method for dynamic modal availability according to claim 2, characterized in that, The driving local model completes low-rank training, generating local low-rank parameters and a unified task representation, including: Generate routing gating vectors based on modal states. And apply it to the visual branch, text branch, and fusion branch, as follows: in, Indicates the visual branch gate value, Indicates the text branch gate value. Indicates the fusion branch gate value; Construct image features respectively Text features and fusion features ,as follows: in, Represents the cross-modal fusion operator; Considering that some clients are in modality-deficient, fusion-modal, or unimodal scenarios, a unified task representation is constructed. ,as follows: in, Indicates client In the round A unified task representation; Local model training is performed based on a unified task representation, with modality-aware local task loss during the training process. The formula is as follows: in, Indicates client In the round The number of local samples, Indicates the first The supervision target for each sample This indicates the currently trainable parameters for this client; To suppress excessive drift caused by local data offset under dynamic modal conditions, a local loss is constructed using proximal constraints on the global model for local updates. This local loss is: in, Represents the proximal regularity coefficient. This represents the set of parameters that are active and allowed to be updated in the current round. Indicates the first A global parameter.

5. The federated multimodal learning adaptive method for dynamic modal availability according to claim 4, characterized in that, The triggering of distillation constraints, representation alignment, and prototype cache replay, updating the current local model parameters, and obtaining the consolidated low-rank parameters include: Teacher output in the complete historical modality Student output in the currently available modalities Distillation constraints are applied to determine distillation losses. ,as follows: in, Indicates the distillation temperature. Indicates the Kullback-Leibler divergence; Establish a representation alignment loss between the fused representation and the single-modal representation. ,as follows: in, Indicates client The fused representation prototype cached in the most recent full modal state; The local loss, distillation loss, and representation alignment loss are combined into a knowledge consolidation objective. ,as follows: in, Indicates the distillation intensity coefficient. This represents the alignment strength coefficient.

6. The federated multimodal learning adaptive method for dynamic modal availability according to claim 5, characterized in that, The modality-aware global aggregation includes: The base weights are calculated based on the number of client samples, and then the stability coefficient is added to obtain the aggregate weights. The formula is as follows: in, Indicates client In the round Local sample size Indicates client In the round stability coefficient, This represents a lower bound constant to prevent the denominator from being zero; The server updates global parameters as follows: in, Indicates client Uploaded partial parameters, This indicates the global parameters for the next round.

7. The federated multimodal learning adaptive method for dynamic modal availability according to claim 6, characterized in that, Also includes: When using FedYogi, FedAdam, FedAdagrad, or FedAvgM adaptive aggregation strategies, the client incrementally writes as Then calculate the modal sensing mean increment. The formula is as follows: in, Indicates the global parameters for the current round. This represents the weighted average increment; When using adaptive server optimization, the server updates global parameters as follows: in, Indicates the server learning rate. Indicates by The obtained first-order momentum term, Represents a second-order statistical term. This represents the numerical stability constant.

8. A federated multimodal learning adaptive system for dynamic modal availability, characterized in that: include: The temporal modal availability modeling module is used to record the modal state of each client, and based on the modal state of the client, to predict the feasibility of the client and obtain the probability of modal availability in the next round. Calculate the switching strength and stability coefficients, and determine the subset of clients participating in this round of training based on the client's next round modality availability probability and stability coefficient; The dynamic architecture routing and local training module is used to determine the modality gating of the visual branch, text branch and fusion branch according to the modality state of each client in the client subset, select the shared adapter or private adapter according to the next round modality availability probability of the client, drive the local model to complete low-rank training, and generate local low-rank parameters and unified task representation. The elastic knowledge consolidation and federated aggregation module is used to determine whether a client has experienced long-term single-modal degradation or high-frequency switching based on the stability coefficient. If so, it triggers distillation constraints, representation alignment, and prototype cache replay, updates the current local model parameters, obtains consolidated low-rank parameters, and uploads the consolidated low-rank parameters to the server. If not, it uploads the local low-rank parameters to the server. The server obtains the updated global parameters based on the modality-aware global aggregation and distributes the updated global parameters to each client participating in the training, thus entering the next federated round.

Images