A risk monitoring-oriented attention integrated multi-modal federated learning method and system

Abstract

This invention discloses an attention-integrated multimodal federated learning method and system for risk monitoring. On the client side, an attention-integrated hybrid early fusion module dynamically injects multimodal features into globally shared latent variables through attention mechanisms and entropy-based gating mechanisms. On the server side, a plug-and-play dual-track federated aggregation strategy decouples state aggregation from model aggregation. The state track employs robust filtering based on the geometric median, while the model track supports algorithm-independent parameter updates. Between communication rounds, a neighborhood-weighted personalized reference mechanism constructs a gradient-based semantic topology to balance global consistency and local task preferences. This invention simultaneously addresses the technical challenges of modality incompleteness, data statistical heterogeneity, and Byzantine adversarial robustness in multimodal federated learning, thus providing an efficient and robust collaborative learning solution for disaster risk perception under privacy-preserving conditions.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to an attention-integrated multimodal federated learning method and system for risk monitoring. Background Technology

[0002] Risk monitoring requires the fusion of multi-source data streams from heterogeneous sensors to achieve comprehensive situational awareness. Federated learning (FL), as a privacy-preserving distributed collaborative training paradigm, has shown significant application prospects in this field. For example, the FedAvg algorithm proposed by McMahan et al. (McMahan, B., Moore, E., Ramage, D., Hampson, S. and y Arcas, BA, 2017, April. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics (pp. 1273-1282). PMLR.) is a foundational work in federated learning. Its core idea is that each client independently performs several rounds of stochastic gradient descent on its local data, and then uploads the updated model parameters to a central server for weighted average aggregation. However, FedAvg is prone to client drift when dealing with highly heterogeneous client data distributions, leading to a decrease in global model convergence performance. To address this, Li et al. (Li, T., Sahu, AK, Zaheer, M., Sanjabi, M., Talwalkar, A. and Smith, V., 2020. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2, pp. 429-450) introduced a proximal regularization term into the local objective function of each client to constrain the deviation between the local and global models, thereby alleviating the drift problem under heterogeneous conditions. Karimireddy et al. (Karimireddy, SP, Kale, S., Mohri, M., Reddi, S., Stich, S. and Suresh, AT, 2020, November. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning (pp. 5132-5143). PMLR.) proposed SCAFFOLD, which theoretically achieves explicit compensation for client drift by introducing control variables to estimate and correct the gradient shift of each client.Furthermore, Wang et al. (Wang, J., Liu, Q., Liang, H., Joshi, G. and Poor, HV, 2020. Tackling the objective inconsistency problem inheterogeneous federated optimization. Advances in neural information processing systems, 33, pp.7611-7623.) proposed FedNova, which eliminates the inconsistency of the objective function caused by the inconsistency of local training rounds by normalizing the number of local update steps of each client. Hsu et al. (Hsu, TMH, Qi, H. and Brown, M., 2019. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335.) proposed FedAvgM, which introduces a momentum mechanism on the server side to smooth global model updates and enhance the stability of the aggregation process under non-independent and identically distributed conditions. Federated learning has shown significant application prospects in fields such as artificial intelligence, healthcare, and the Internet of Things.

[0003] Multimodal fusion (MF) refers to the technique of integrating information from different perceptual channels into a unified semantic space, aiming to improve the model's perception and reasoning capabilities by leveraging the complementarity between modalities. Personalized Federated Learning (PFL) aims to alleviate the problem in federated learning where a "one-size-fits-all" global model cannot adapt to the local tasks of each client due to data heterogeneity. Existing methods, such as FedRep and FedProx proposed by Collins et al. (Collins, L., Hassani, H., Mokhtari, A. and Shakkottai, S., 2021, July. Exploiting shared representations for personalized federated learning. In International conference on machine learning (pp. 2089-2099). PMLR.), explicitly divide the model into a globally shared representation layer and client-specific prediction heads. Only the representation layer parameters are shared for global aggregation, while the prediction heads are trained independently locally. While the above methods have made significant progress in mitigating statistical heterogeneity, they generally assume that each client has complete modal inputs, and their ability to accurately align semantics is insufficient when faced with complex scenarios where incomplete modal inputs and statistical heterogeneity coexist.

[0004] Byzantine defense strategies target malicious actors injecting poisoned gradients or model parameters into federated learning systems. Since servers cannot directly access local data, malicious clients can manipulate the global model by uploading carefully crafted fake updates. Existing defense strategies, such as Krum proposed by Blanchard et al. (Blanchard, P., El Mhamdi, EM, Guerraoui, R. and Stainer, J., 2017. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in neural information processing systems, 30.), select the single gradient with the smallest sum of Euclidean distances to other updates as the aggregation result, thus excluding anomalous updates that deviate from the group. However, these methods are primarily designed for unimodal scenarios and do not consider the gradient sparsity and norm fluctuations caused by modal heterogeneity in multimodal federated learning, thus limiting their effectiveness in multimodal environments. Harmony, proposed by Ouyang et al. (Ouyang, X., Xie, Z., Fu, H., Cheng, S., Pan, L., Ling, N., Xing, G., Zhou, J. and Huang, J., 2023, June. Harmony: Heterogeneous multi-modal federated learning through disentangled model training. In Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services (pp. 530-543).), is a representative multimodal federated learning method that achieves modality alignment and fusion by utilizing a Transformer architecture for cross-modal attention interaction at the input layer. While the aforementioned works have made progress in cross-modal fusion, their global aggregation performance significantly degrades when faced with severe statistical heterogeneity, and they lack an effective mechanism for maintaining semantic consistency under modal incompleteness.

[0005] In summary, the existing technology has the following key shortcomings:

[0006] (1) Multimodal federated learning methods lack robustness to non-independent and identically distributed (i.i.d.) data. Existing MFL methods focus on the design of cross-modal fusion strategies, aiming to achieve information complementarity and semantic alignment between modalities at the feature layer or decision layer. However, these methods usually assume that the data distribution of each client is approximately consistent or has limited deviation when designing the aggregation mechanism. In actual monitoring scenarios, data collected in different regions and at different times have significant differences in category distribution, constituting a serious non-independent and identically distributed condition. Under this condition, the local update direction of each client is quite divergent, and simple weighted average aggregation will cause the global model to shift towards the compromise direction of different local optima, thus failing to effectively converge to a global representation that has good generalization ability for all clients. At the same time, the cross-modal fusion module implicitly relies on the statistical regularity of the training data in the feature alignment process. When the data distribution shifts drastically, the attention weights and feature mapping relationships learned by the fusion layer are difficult to transfer between different clients, resulting in an irreconcilable contradiction between the semantic alignment accuracy and the local task adaptability of the global model.

[0007] (2) Personalized federated learning methods assume modality integrity. Existing PFL methods have made significant progress in dealing with statistical heterogeneity. Strategies such as proximal constraints, parameter decoupling, and cluster comparison can balance global consensus and local preferences to some extent. However, these methods generally assume that all clients have complete and consistent modal inputs in their model architecture and optimization process design. In actual monitoring networks, edge devices have varying sensor configurations, communication links are extremely unstable in harsh environments, and factors such as equipment failure and energy depletion can cause some clients to only provide data in a single modality during specific periods. When the proportion of missing modalities is high, the fusion module designed based on complete modal inputs will exhibit undefined behavior or significant performance degradation due to missing input dimensions. The personalization mechanism of PFL methods does not have the ability to perceive and compensate for modal missingness, making it difficult for the entire system to maintain effective collaborative learning under conditions of incomplete modality and statistical heterogeneity.

[0008] (3) Existing Byzantine defense methods are not adapted to multimodal federated learning scenarios. Federated learning systems are inherently vulnerable to poisoning attacks by malicious actors because the server cannot directly access the client's local data. Although existing Byzantine defense strategies have shown some defensive effectiveness in single-modal federated learning, these methods are all designed for single-modal, homogeneous model structures. In multimodal federated learning, different clients have different modal combinations, resulting in inherent differences in the dimension, gradient norm, and sparsity of the uploaded model updates. This fluctuation in gradient statistical features caused by modal heterogeneity is easily misjudged as abnormal behavior, leading to a high false positive rate in the defense mechanism and incorrectly identifying normal clients with different modal configurations as malicious actors. At the same time, under highly Non-IID conditions, the gradient directions of legitimate clients are inherently highly dispersed, which further blurs the boundary between normal updates and malicious poisoning updates, significantly reducing the filtering accuracy of existing defense methods. Summary of the Invention

[0009] To address the key deficiencies of the prior art, this invention proposes an attention-integrated multimodal federated learning method and system for risk monitoring. It aims to provide an efficient and robust distributed multimodal collaborative learning scheme for risk monitoring under privacy protection conditions, while solving the three major technical challenges of modal incompleteness, data statistical heterogeneity, and Byzantine poisoning attacks in the monitoring federated learning scenario.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] An attention-integrated multimodal federated learning method for risk monitoring includes the following steps:

[0012] In each communication round, the server broadcasts the current global latent variable state to the clients participating in the training and sends each client its own personalized reference signal.

[0013] The client receives the global latent variable state and personalized reference signal, and dynamically fuses the local multimodal data into a unified latent variable space through the attention integration hybrid early fusion module to generate a local fusion state; it then performs local task training based on the local fusion state to obtain local model parameter updates and local state updates, and uploads them to the server.

[0014] The server receives local state updates and local model parameter updates uploaded by each client, and decouples and aggregates them through a plug-and-play dual-track federated aggregation strategy. The state track performs robust aggregation of all local state updates based on the geometric median to update the global latent variable state. The model track performs independent aggregation of the model parameters corresponding to each mode through pluggable aggregation operators.

[0015] Based on the updated global latent variable state and the local state updates of each client, the server constructs a semantic topology through a neighborhood-weighted personalized reference mechanism, and synthesizes a personalized reference signal for the next communication round for each client.

[0016] Repeat the above steps until the model converges, ultimately generating a personalized model for each client.

[0017] Furthermore, the attention-integrated hybrid early fusion module performs the following steps:

[0018] The global latent variable state issued by the server is used as the initialization starting point for local fusion, and layer normalization is performed on the input features of latent variables and available modalities respectively;

[0019] In each fusion layer, the normalized latent variables are used as queries, and the normalized modal features are used as keys and values. Cross-attention calculation is performed to obtain the context vector of each modality.

[0020] Calculate the normalized entropy of the cross-attention weight matrix for each modality, and generate adaptive gating coefficients based on the entropy value; when a modality is missing or the signal quality is low, the entropy value of the attention distribution increases, the gating coefficients approach zero, and the contribution of that modality is automatically suppressed.

[0021] The context vector is modulated using gating coefficients, and the latent variables are updated through a self-normalized feedforward network;

[0022] After multi-layer iterative fusion, cross-modal information is integrated through a self-attention layer to generate the final local fusion state.

[0023] Furthermore, during local training, the client's loss function includes task prediction loss and a personalization regularization term; the personalization regularization term is used to constrain the difference between the current local fusion state and the received personalized reference signal in order to balance global consistency and local task preference.

[0024] Furthermore, in the plug-and-play dual-track federated aggregation strategy, the aggregation of state tracks specifically includes:

[0025] Calculate the geometric median of local state updates uploaded by all clients as an estimate of the central tendency of the population updates;

[0026] Calculate the cosine similarity between each client's local state update and the geometric median;

[0027] The cosine similarity is normalized to generate adaptive aggregation weights for each client.

[0028] All local state updates are weighted and aggregated based on adaptive aggregation weights, and combined with the global state of the current round to obtain the updated global latent variable state.

[0029] Furthermore, in the plug-and-play dual-track federated aggregation strategy, the aggregation of model tracks specifically includes:

[0030] For each modality, identify the subset of active clients that possess that modality in the current communication round;

[0031] If the subset is not empty, the preset federated aggregation operator is called to aggregate the model parameters corresponding to the modality uploaded by each client in the subset and update the parameters corresponding to the modality in the global model.

[0032] If the subset is empty, the parameters of that modality in the global model remain unchanged.

[0033] Furthermore, the federated aggregation operator is an algorithm-independent plug-and-play interface that can be adapted to any of the federated optimization algorithms: FedAvg, FedNova, SCAFFOLD, FedAdam, and FedAvgM.

[0034] Furthermore, the neighborhood-weighted personalized reference mechanism specifically includes:

[0035] Based on the local state updates uploaded by all clients, calculate the cosine similarity between any two clients and construct a pairwise similarity matrix;

[0036] For the target client, its similarity vector with all other clients is min-max normalized to amplify the distinguishability of neighboring clients;

[0037] Apply the softmax function to the normalized vector to obtain the neighborhood weight coefficients of the target client relative to other clients;

[0038] Based on the neighborhood weight coefficient, the local state updates of other clients are weighted and summed, and combined with the updated global state, to synthesize the personalized reference signal of the target client in the next communication round.

[0039] Secondly, the present invention provides an attention-integrated multimodal federated learning system for risk monitoring, comprising a server and multiple clients, the system being used to perform the method described in any of the above-mentioned embodiments.

[0040] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0041] Fourthly, the present invention provides a client device for multimodal federated learning, comprising:

[0042] The communication module is used to receive the global latent variable state and the unique personalized reference signal broadcast by the server, and to upload the local state update and local model parameter update to the server.

[0043] An attention-integrated hybrid early fusion module is used to perform the steps of the method as described in claim 2, fusing local multimodal data into a local fusion state;

[0044] The training module is used to calculate the loss function based on the local fusion state and the personalized reference signal, and update the local model parameters to generate the local state update and the local model parameter update.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] Regarding multimodal fusion capabilities, existing personalized federated learning methods such as FedAvg, FedProx, and FedRep lack multimodal fusion functionality, performing only federated optimization for single-modal inputs. While Harmony supports cross-modal interaction based on Transformer, it lacks a dedicated mechanism for handling modality loss. This invention achieves deep multimodal fusion with adaptive handling of modality loss by combining latent variables, cross-attention, and entropy gating.

[0047] In addressing statistical heterogeneity, FedAvg lacks a specific mechanism and is prone to client drift under non-independent and identically distributed conditions. FedProx provides limited drift mitigation through proximal constraints, but it uses the global model as the sole reference point and cannot distinguish the task similarity between different clients. FedRep achieves representation layer sharing and prediction head privatization through parameter decoupling, but lacks selective knowledge sharing between clients. Methods such as Harmony and Krum lack dedicated mechanisms for dealing with statistical heterogeneity. This invention addresses statistical heterogeneity through a three-layer mechanism: regularization constraints, global latent variable semantic anchoring, and neighborhood-weighted personalized reference.

[0048] Regarding Byzantine defense capabilities, FedAvg, FedProx, FedRep, and Harmony lack any Byzantine defense capabilities. While methods such as Krum and pruned mean support Byzantine robust aggregation, they are designed for updating unimodal isomorphic models, and their defense accuracy is limited in multimodal heterogeneous scenarios due to inconsistent parameter spaces. This invention establishes robust aggregation on a unified latent variable semantic space, fundamentally avoiding the interference of modal heterogeneity on anomaly detection.

[0049] Regarding optimizer flexibility, existing methods are typically tied to specific optimization strategies, requiring redesign of the aggregation logic when changing optimizers. This invention, through a plug-and-play interface design for model tracks, completely decouples the fusion and defense mechanisms from the underlying optimizer, enabling seamless adaptation to various mainstream federated optimization algorithms.

[0050] In summary, existing methods only cover one or a few of the aforementioned technical dimensions, while this invention achieves comprehensive coverage across multiple dimensions, including multimodal fusion, handling of missing modalities, addressing statistical heterogeneity, Byzantine defense, personalization capabilities, and optimizer flexibility. Its specific advantages are as follows:

[0051] (1) Robustness under modality missing conditions is significantly improved. The adaptive gating mechanism based on attention entropy can automatically perceive the information availability and quality level of each modality without any prior missing information. When a modality is normally available, its attention distribution exhibits a focused pattern, and the corresponding gating value approaches 1, fully utilizing the semantic contribution of that modality. When a modality is missing or the signal degrades, the attention distribution tends to be uniform, and the gating value automatically approaches 0, effectively blocking the interference of missing modalities on the fusion process. This mechanism does not require the design of a dedicated missing modality detection module or completion branch, avoiding additional computational overhead and engineering complexity.

[0052] (2) Strong robustness to statistical heterogeneity. This invention addresses the challenge of statistical heterogeneity through a multi-layered mechanism. On the client side, globally shared latent variables serve as a unified semantic anchor, providing a consistent reference benchmark for the fusion process of all clients and mitigating the feature space misalignment problem caused by differences in data distribution. The regularization term in the local training loss constrains the deviation between the personalized state and the global state, preventing the local model from deviating excessively from the global consensus. On the server side, the state trajectory suppresses the excessive influence of client updates with severe distribution shifts on the global state through geometric median and cosine reweighting. The neighborhood-weighted personalized reference mechanism alleviates the heterogeneity problem from another dimension by allowing clients to selectively refer to the learning experience of neighbors with similar optimization directions, rather than forcing all clients to converge to a single global model.

[0053] (3) Strong robustness against Byzantine poisoning attacks. The geometric median estimation in the state orbit is inherently robust to outliers. Even with a significant proportion of malicious clients uploading carefully crafted poisoned updates, the geometric median tends to be located near the clustering region of normal updates. Furthermore, cosine similarity reweighting amplifies the contribution of normal updates and compresses the impact of malicious updates. The entire defense mechanism is embedded within the aggregation process, requiring no separate attack detection stage or additional trusted datasets, and no pre-set threshold for the proportion of malicious clients. Even under severe adversarial conditions with a very high proportion of malicious clients, the system's performance retention rate is significantly better than existing baseline methods, demonstrating excellent adversarial robustness.

[0054] (4) Optimizer-independent plug-and-play enhancement capabilities. The standardized aggregation interface design of the model orbits allows the fusion method to be seamlessly integrated into various mainstream federated optimization algorithms as an enhancement module without modifying the internal logic of each algorithm. The relative performance gain ratio of the method to different underlying optimizers is highly consistent, and the performance fluctuation between various PAHE variants is minimal. The plug-and-play feature makes this framework highly practical for engineering applications. Users can flexibly select the underlying optimizer according to specific application scenarios and resource constraints without redesigning or tuning the fusion and defense mechanisms. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0056] Figure 1 This is a schematic diagram of the overall architecture of the attention-integrated multimodal federated learning method for risk monitoring provided in an embodiment of the present invention.

[0057] Figure 2 An internal flowchart of the attention integration hybrid early fusion module provided in an embodiment of the present invention.

[0058] Figure 3 This is a schematic diagram of a single fusion layer structure provided in an embodiment of the present invention.

[0059] Figure 4 This is a schematic diagram of a plug-and-play dual-track multimodal federated aggregation strategy provided in an embodiment of the present invention.

[0060] Figure 5 This is a schematic diagram of the neighborhood-weighted personalized reference mechanism provided in an embodiment of the present invention. Detailed Implementation

[0061] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0062] The attention-integrated multimodal federated learning method for risk monitoring proposed in this invention has the following overall architecture: Figure 1 As shown, the framework consists of three core components: an attention-integrated hybrid early fusion module on the client side, a plug-and-play dual-track federated aggregation strategy on the server side, and a neighborhood-weighted personalized reference mechanism between communication rounds.

[0063] On the client side, the attention-integrated hybrid early fusion module dynamically injects multimodal features into globally shared latent variables through attention mechanisms and entropy-based gating mechanisms. On the server side, a plug-and-play dual-track federated aggregation strategy decouples state aggregation from model aggregation. State tracks employ robust filtering based on geometric median, while model tracks support algorithm-independent parameter updates. Between communication rounds, a neighborhood-weighted personalized reference mechanism constructs a gradient-based semantic topology to balance global consistency and local task preferences. This invention includes the following technical objectives:

[0064] (1) Achieving robust multimodal semantic fusion under modality missing conditions. A hybrid early fusion module based on attention mechanism is designed, introducing globally shared latent variables as cross-modal semantic anchors, and iteratively absorbing feature information of each available modality through cross-attention mechanism. By introducing an adaptive gating mechanism based on attention distribution entropy, the fusion process can automatically perceive the information quality of each modality input, fully utilize its semantic contribution when the modality is normally available, and automatically suppress its influence when the modality is missing or the signal quality is low, thus maintaining robust multimodal representation without explicit missing labeling or additional missing processing procedures.

[0065] (2) Constructing a federated aggregation mechanism that combines Byzantine robustness and optimizer flexibility. A plug-and-play dual-track federated aggregation strategy is proposed, decoupling the robust aggregation of semantic states from the optimization aggregation of modality-specific model parameters into two independent processing tracks. The state track filters malicious client poisoning updates through robust estimation based on geometric median and a cosine similarity reweighting mechanism, achieving adaptive defense against Byzantine attacks without relying on trusted datasets. The model track provides a standardized plug-and-play aggregation interface, enabling the framework to seamlessly adapt to various mainstream federated optimization algorithms without requiring any modification to the robust aggregation logic, thus balancing system security and optimization flexibility.

[0066] (3) Realize personalized knowledge transfer based on semantic topology. A neighborhood-weighted personalized reference mechanism is introduced to construct a semantic similarity topology graph based on gradient direction from the state update vectors of each client, automatically identifying client neighbors with similar task features and optimization directions. Based on this topology, a weighted personalized reference signal is synthesized for each client, enabling each client to selectively benefit from the learning experience of semantic neighbors in subsequent training, rather than relying solely on a single global average model. This mechanism allows the framework to maintain global semantic consistency while fully respecting and adapting to the local task preferences and data distribution characteristics of each client, effectively balancing global generalization ability and local adaptation accuracy.

[0067] 1. Attention-integrated hybrid early fusion module

[0068] To effectively fuse multimodal heterogeneous data into a unified semantic space, this invention proposes an attention-integrated hybrid early fusion module, such as... Figure 2 As shown, the module employs an iterative latent variable update mechanism, using attention and entropy-driven gating to achieve dynamic feature selection and injection. In the... Communication rounds, client First, receive the global latent variable parameters broadcast by the server. And use it as the initial starting point for local integration. To mitigate gradient instability caused by differences in the numerical distribution of pre-trained features, the client... modality enter With current latent variables implement:

[0069]

[0070]

[0071] in, and These represent the normalized modal input features and the current latent variable state, respectively. To handle high-dimensional modal inputs, this invention actively queries the input features using a small number of latent variables. For modal... First, through the projection matrix Calculate query ,key AND value :

[0072]

[0073] in, The sequence length is fixed as the number of latent variables. , and input length Decoupling. Modal context is computed by scaling dot product attention and utilizing the output projection matrix. Map it back to the latent variable space:

[0074]

[0075] in, Represents the attention weight matrix. Scaling factor This is the aggregated modal context vector, and This is responsible for projecting the context dimension back to the latent variable dimension. .

[0076] To automatically suppress low-quality modes, this invention introduces a gating mechanism based on information entropy. The attention distribution is calculated. row average normalized entropy And based on this, scalar gating coefficients are generated. :

[0077]

[0078]

[0079] in, To prevent numerically unstable minimal constants, This represents the activation function. This is the temperature coefficient. When modes are missing, the attention distribution tends to be uniform, the entropy value increases, leading to... This prevents the mode from updating the latent variables.

[0080] Each fusion layer first calculates the joint modal contribution, and then applies an entropy gate to control the entire update process:

[0081]

[0082] in, This indicates a self-normalized feedforward network employing the SELU mechanism.

[0083] When a certain mode is missing This will simultaneously suppress cross-attention context and related nonlinear transformations. For example... Figure 4 As shown, after the fusion layer After several iterations, the final self-attention layer integrates cross-modal information and generates a local fusion state. :

[0084]

[0085] The loss function for local training consists of task prediction loss and a personalization regularization term, defined as follows:

[0086]

[0087] in, For a specific task, the loss function is... For predicting the head, For real labels, The regularization coefficient is . The norm is used to constrain the local personalized state from deviating too far from the global consensus. The code for the attention-integrated hybrid early fusion module is shown in Algorithm 1.

[0088]

[0089]

[0090] This invention proposes a novel multimodal fusion architecture based on a globally shared latent variable and entropy gating attention integration hybrid early fusion mechanism. First, a set of globally shared latent variables is introduced as cross-modal semantic anchors. These latent variables are uniformly maintained by the server and distributed to each client at the start of each communication round, providing a unified semantic alignment reference for all clients. Furthermore, the fusion process uses latent variables as queries and modal features as key-value pairs for cross-attention calculation. The query dimension is fixed to the number of latent variables and is independent of the sequence length of each modal input, thus giving the fusion process inherent structural robustness to differences in feature dimensions and sequence lengths across different modalities. Further, an adaptive gating mechanism based on attention distribution entropy is introduced, quantifying the degree of focus of modal information by calculating the normalized entropy of the attention weight matrix of each modality. When a modality is missing or the signal quality is low, the attention distribution tends to be uniform, resulting in a high entropy value, and the corresponding gating value automatically approaches zero. The method automatically suppresses the cross-attention context output of the missing modality and its associated nonlinear transformation path, achieving implicit awareness and automatic masking of missing modalities without the need for additional missing labeling or explicit missing processing branches. Finally, the fusion layer employs a self-normalized feedforward network combined with layer normalization to maintain gradient stability during multiple iterative fusion processes. After L iterations of the fusion layer, the final self-attention layer integrates cross-modal information to generate a compact local fusion state representation. This invention enables the system to maintain effective multimodal semantic representation even under extreme conditions of severe modality loss.

[0091] 2. Plug-and-play dual-track multimodal federated aggregation strategy

[0092] To address the challenges of statistical heterogeneity and modality loss in multimodal federated learning, the server-side employs a plug-and-play dual-track multimodal federated aggregation strategy mechanism, such as... Figure 5As shown. This mechanism decouples the information flow between the state space and the model space, executing targeted state and model aggregation strategies separately. In the state aggregation orbit, the latent variable state update vector uploaded by the client is processed. The present invention first seeks a geometric median vector that can represent the central trend of all uploaded updates. Its optimization objective is defined as:

[0093]

[0094] in, This represents the center vector to be solved. For the first Local status updates uploaded by each client. The norm is used to represent the reliability of each client update. To further quantify the reliability of each client update, this invention calculates the norm for each upload vector. Geometric Mean Cosine similarity between :

[0095]

[0096] in, Reflects the client The similarity metric measures the degree of consistency between the client's update direction and the group consensus; a higher value indicates a more reliable update from that client. Based on this similarity metric, we assign adaptive aggregation weights to each client. The weighted update vector is then used to generate the global state for the next round. :

[0097]

[0098] in, This represents the global state in the current round. These are normalized adaptive weights. The reweighting strategy effectively guides the evolution of the global state to the semantic consensus region, significantly reducing the risk of model drift caused by data heterogeneity.

[0099] In the model aggregation track, this invention leverages the algorithm independence of aggregation operators to design a plug-and-play modular aggregation mechanism. This mechanism provides a universal parameter update interface, allowing for the flexible integration or replacement of different federated aggregation algorithms without modifying the underlying communication protocol or model architecture. For any modality... The server identifies a subset of active clients with that modality in the current communication round. Invoke the selected federated aggregation operator. Perform parameter updates on the projection matrix within this set:

[0100]

[0101] in, Represents the first after aggregation Modal projection parameters, For the client The uploaded corresponding modal parameters, This provides auxiliary information such as sample size. This represents pluggable federated aggregation functions, ensuring that the global projection layer can learn general feature extraction capabilities from all available data, unaffected by modality-deficient clients. Meanwhile, By decoupling at the aggregation algorithm level, plug-and-play functionality is achieved, flexibly adapting to dynamically changing heterogeneous modal configurations and diverse optimization needs. The plug-and-play dual-track multimodal federated aggregation strategy code is shown in Algorithm 2.

[0102]

[0103]

[0104] The plug-and-play dual-track federated aggregation strategy proposed in this invention is a server-side aggregation architecture that decouples the aggregation process into two functionally independent processing tracks. First, the state track is dedicated to robust updates of global latent variables, calculating the geometric median of the state update vectors uploaded by all clients to obtain an estimate of the central tendency of the updates. It calculates the cosine similarity between each update and the geometric median, and converts this similarity into adaptive aggregation weights through exponentially weighted softmax normalization, giving higher weights to updates aligned with the group consensus direction. Abnormal updates deviating from the group direction are naturally downweighted. The entire process does not require a pre-set malicious client proportion threshold or reliance on trusted verification datasets. Second, the model track is responsible for aggregating modality-specific model parameters, providing a standardized plug-and-play aggregation interface. It independently identifies a subset of active clients possessing that modality and applies the selected aggregation operator. This interface is compatible with various mainstream federated optimization algorithms such as FedAvg, FedNova, SCAFFOLD, FedAdam, and FedAvgM. Switching optimizers only requires replacing the aggregation operators without modifying the robust aggregation logic of the state track. Ultimately, the decoupled design of the two tracks ensures the orthogonality between robustness and optimization flexibility, meaning that enhanced Byzantine defense does not limit the choice of optimization strategy. This invention simultaneously achieves robust defense against Byzantine poisoning attacks and seamless adaptation to various federated optimization algorithms.

[0105] 3. Neighborhood-weighted personalized reference mechanism

[0106] To further facilitate implicit knowledge transfer between clients performing similar tasks, this invention introduces a neighborhood-weighted personalized reference mechanism on the server side, such as... Figure 5 As shown. This mechanism utilizes gradient update information uploaded by the client to construct a semantic topology, and based on this, customizes the guiding objective for the next round of optimization for each client. First, the server constructs a semantic adjacency matrix based on the update vector obtained from the state aggregation orbit. For any two clients... and Their similarity Defined as:

[0107]

[0108] in, and They represent the client respectively. and Uploaded local state update vector, The consistency between the two in the direction of gradient optimization was quantified.

[0109] To enhance the influence of similar neighbors and suppress interference from irrelevant clients, this invention normalizes the similarity distribution and calculates the original neighborhood weights. :

[0110]

[0111] in, and They represent the client respectively. The maximum and minimum similarity with all other clients, To prevent numerical stability constants with a denominator of zero, the following will be used: Mapped to normalized aggregation coefficients This is to ensure the probability distribution characteristics of the weights.

[0112] Finally, using this neighborhood weight, the server acts as a client. Synthesis of the first wheel's personalized reference status The reference state is determined by, except for The latest estimated state from other clients is aggregated in a weighted manner:

[0113]

[0114] in, This represents the current global state. Representing the neighbors Final state estimation after local training. It will be distributed to clients in the next round. This serves as the regularization anchor point in its local personalization updates. While maintaining local personalized features, the client selectively absorbs beneficial knowledge from similar clients within the semantic neighborhood, significantly improving the model's generalization performance in Non-IID scenarios. The code for the neighborhood-weighted personalization reference mechanism is shown in Algorithm 3.

[0115]

[0116] This invention provides a neighborhood-weighted personalized reference mechanism based on gradient semantic topology, which is a personalized mechanism for selective knowledge transfer by constructing a semantic similarity topology graph among clients. First, based on the state update vectors uploaded by each client in its state orbit, the cosine similarity between all client pairs is calculated to construct a pairwise similarity matrix reflecting the similarity of client optimization directions. Furthermore, the similarity vector of each client is min-max normalized to amplify the discriminative power of nearest neighbors, and then mapped to normalized neighborhood weight coefficients using softmax, giving higher weights to clients with similar semantic directions. Further, these weight coefficients are used to weighted synthesize the estimated post-training states of all neighboring clients, generating a personalized reference signal for each client, which is distributed to the corresponding client at the start of the next round of communication. Finally, during local training, the client weighted aggregates this personalized reference signal with its own historical state and the current local fusion result, ensuring that each client not only acquires general knowledge from the global model. This invention effectively bridges the gap between global generalization and local adaptation, significantly suppressing performance degradation under extreme statistical heterogeneity conditions.

[0117] 4. Overall Training Process

[0118] This invention provides an attention-integrated multimodal federated learning method for risk monitoring. It comprehensively utilizes key mechanisms such as an attention-integrated hybrid early fusion module, a plug-and-play dual-track multimodal federated aggregation strategy, and neighborhood-weighted personalized references to achieve deep cross-modal semantic fusion and robust modeling in complex distributed scenarios with modality loss and Non-IID. By introducing an attention-integrated hybrid early fusion module on the client side, high-dimensional heterogeneous modal features are injected into a unified low-dimensional latent variable space. The client side significantly improves the numerical stability and noise resistance of deep feature extraction using a gated SELU-based SNN. Combined with the server-side dual-track aggregation strategy, this method first effectively defends against model drift caused by statistical heterogeneity through state track geometric median and adaptive weighting mechanisms. Furthermore, the framework successfully achieves flexible adaptation to dynamic modality configurations and implicit knowledge transfer across clients through a plug-and-play model track interface and a neighborhood reference generation mechanism. Finally, this method has the ability to construct a highly robust, scalable, and highly generalizable distributed multimodal federated learning system. The overall training process code is shown in Algorithm 4.

[0119]

[0120]

[0121] In summary, the attention-integrated hybrid early fusion module, the plug-and-play dual-track multimodal federated aggregation strategy, and the neighborhood-weighted personalized reference mechanism proposed in this invention—a unified framework design with three components working collaboratively—is not a simple combination of isolated modules, but rather achieves organic integration and collaborative optimization through a shared latent variable state space. The robust state representation generated by entropy-gated fusion provides a more stable input distribution for estimating the geometric median of state orbits, reducing the risk of normal client updates being misjudged as abnormal. The clean global state generated after filtering malicious updates from state orbits, in turn, provides more reliable semantic anchors for the fusion modules of each client. The personalized reference mechanism constructs the topology based on the filtered client updates, avoiding the pollution of neighborhood relationships by malicious clients. This collaborative design makes the overall framework performance superior to the effect of each component working independently or simply cascading.

[0122] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An attention-integrated multimodal federated learning method for risk monitoring, characterized in that, Includes the following steps: In each communication round, the server broadcasts the current global latent variable state to the clients participating in the training and sends each client its own personalized reference signal. The client receives the global latent variable state and personalized reference signal, and dynamically fuses the local multimodal data into a unified latent variable space through the attention integration hybrid early fusion module to generate a local fusion state; it then performs local task training based on the local fusion state to obtain local model parameter updates and local state updates, and uploads them to the server. The server receives local state updates and local model parameter updates uploaded by each client, and decouples and aggregates them through a plug-and-play dual-track federated aggregation strategy. The state track performs robust aggregation of all local state updates based on the geometric median to update the global latent variable state. The model track performs independent aggregation of the model parameters corresponding to each mode through pluggable aggregation operators. Based on the updated global latent variable state and the local state updates of each client, the server constructs a semantic topology through a neighborhood-weighted personalized reference mechanism, and synthesizes a personalized reference signal for the next communication round for each client. Repeat the above steps until the model converges, ultimately generating a personalized model for each client.

2. The attention-integrated multimodal federated learning method for risk monitoring according to claim 1, characterized in that, The attention-integrated hybrid early fusion module performs the following steps: The global latent variable state issued by the server is used as the initialization starting point for local fusion, and layer normalization is performed on the input features of latent variables and available modalities respectively; In each fusion layer, the normalized latent variables are used as queries, and the normalized modal features are used as keys and values. Cross-attention calculation is performed to obtain the context vector of each modality. Calculate the normalized entropy of the cross-attention weight matrix for each modality, and generate adaptive gating coefficients based on the entropy value; When a mode is missing or the signal quality is poor, the attention distribution entropy increases and the gating coefficient approaches zero, automatically suppressing the contribution of that mode. The context vector is modulated using gating coefficients, and the latent variables are updated through a self-normalized feedforward network; After multi-layer iterative fusion, cross-modal information is integrated through a self-attention layer to generate the final local fusion state.

3. The attention-integrated multimodal federated learning method for risk monitoring according to claim 2, characterized in that, When the client trains locally, its loss function includes task prediction loss and a personalization regularization term. The personalization regularization term is used to constrain the difference between the current local fusion state and the received personalization reference signal in order to balance global consistency and local task preference.

4. The attention-integrated multimodal federated learning method for risk monitoring according to claim 1, characterized in that, In the plug-and-play dual-track federated aggregation strategy, the aggregation of state tracks specifically includes: Calculate the geometric median of local state updates uploaded by all clients as an estimate of the central tendency of the population updates; Calculate the cosine similarity between each client's local state update and the geometric median; The cosine similarity is normalized to generate adaptive aggregation weights for each client. All local state updates are weighted and aggregated based on adaptive aggregation weights, and combined with the global state of the current round to obtain the updated global latent variable state.

5. The attention-integrated multimodal federated learning method for risk monitoring according to claim 1 or 4, characterized in that, In the plug-and-play dual-track federated aggregation strategy, the aggregation of model tracks specifically includes: For each modality, identify the subset of active clients that possess that modality in the current communication round; If the subset is not empty, the preset federated aggregation operator is called to aggregate the model parameters corresponding to the modality uploaded by each client in the subset and update the parameters corresponding to the modality in the global model. If the subset is empty, the parameters of that modality in the global model remain unchanged.

6. The attention-integrated multimodal federated learning method for risk monitoring according to claim 5, characterized in that, The federated aggregation operator is an algorithm-independent plug-and-play interface that can be adapted to any of the federated optimization algorithms: FedAvg, FedNova, SCAFFOLD, FedAdam, and FedAvgM.

7. The attention-integrated multimodal federated learning method for risk monitoring according to claim 1, characterized in that, The neighborhood-weighted personalized reference mechanism specifically includes: Based on the local state updates uploaded by all clients, calculate the cosine similarity between any two clients and construct a pairwise similarity matrix; For the target client, its similarity vector with all other clients is min-max normalized to amplify the distinguishability of neighboring clients; Apply the softmax function to the normalized vector to obtain the neighborhood weight coefficients of the target client relative to other clients; Based on the neighborhood weight coefficient, the local state updates of other clients are weighted and summed, and combined with the updated global state, to synthesize the personalized reference signal of the target client in the next communication round.

8. An attention-integrated multimodal federated learning system for risk monitoring, characterized in that, The system includes a server and multiple clients, and is used to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

10. A client device for multimodal federated learning, characterized in that, include: The communication module is used to receive the global latent variable state and the unique personalized reference signal broadcast by the server, and to upload the local state update and local model parameter update to the server. An attention-integrated hybrid early fusion module is used to perform the steps of the method as described in claim 2, fusing local multimodal data into a local fusion state; The training module is used to calculate the loss function based on the local fusion state and the personalized reference signal, and update the local model parameters to generate the local state update and the local model parameter update.

Images