Multi-modal federal learning method based on self-attention

Abstract

The invention discloses a multi-modal federated learning method based on self-attention, and relates to the field of data processing, and the method comprises the steps: firstly, designing a multi-modal feature coding module at a client, carrying out the deep semantic modeling of image and text modals based on a Transform architecture, and capturing the internal structure information of each modality; secondly, a local cross-modal self-attention modeling module is introduced, fine-grained semantic alignment between images and texts is modeled through a bidirectional cross-attention mechanism, and the local cross-modal understanding ability is enhanced; a multi-head collaborative fusion optimization module is further combined, inter-modal deep semantic collaborative modeling is carried out under a multi-attention perspective, and a fusion effect is optimized through a gating mechanism, so that the problems of inter-modal conflicts and inconsistent expressions are effectively relieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-modal federated learning method based on self-attention. BACKGROUND

[0002] In recent years, federated learning has shown broad application prospects in the presence of real-world problems such as data silos and communication restrictions due to its distributed training capabilities. At the same time, with the widespread existence of multi-modal data such as images and text, multi-modal federated learning, as an emerging research direction, is gradually becoming a research hotspot in the field of computer vision and natural language processing integration. Multi-modal tasks such as image-text matching, image annotation, and image-text retrieval are widely used in cross-modal search, auxiliary blind reading, and social media analysis.

[0003] In multi-modal learning, how to effectively fuse information between heterogeneous modalities to achieve semantic alignment between modalities is the key to improving model performance. In recent years, the self-attention mechanism has made significant breakthroughs in this direction, especially in cross-modal feature modeling. For example, pre-training models such as UNITER and ViLBERT introduce multi-layer self-attention modules to model deep interactions between images and text, thereby improving semantic understanding capabilities. However, in traditional centralized training architectures, the training of such large models relies on massive centralized data and high-performance computing power, which is not suitable for learning tasks in a multi-party heterogeneous distributed environment.

[0004] Traditional multi-modal federated learning often relies on static aggregation strategies and single-modal modeling in the presence of data heterogeneity between clients and semantic alignment between modalities, making it difficult to fully exploit cross-modal high-order correlations between images and text, resulting in limited model generalization and expression capabilities. In addition, existing methods often ignore the feature drift caused by dynamic changes in modal differences, lacking modeling of local and global multi-granularity semantic relationships. SUMMARY

[0005] The present application aims to provide a multi-modal federated learning method based on self-attention to solve the problems raised in the background.

[0006] The technical solution of the present application is: a multi-modal federated learning method based on self-attention, comprising the following steps: Step S1, design a multi-modal feature encoding module on the client side, then use an image encoder and a text encoder to represent image modal and text modal data, respectively, both based on the Transformer backbone structure of the self-attention mechanism to capture deep feature semantics within each modality; Step S2, introduce a local cross-modal self-attention modeling module, through multiple rounds of local training for each client, use intra-modal self-attention and inter-modal cross-attention to extract multi-level and high semantic local multi-modal features, and then model the fine-grained semantic alignment between text and image through a bidirectional cross-attention mechanism; Step S3, combine a multi-head collaborative fusion optimization module, the locally uploaded multi-modal features are respectively sent into multiple federated attention heads to construct a multi-view global representation, perform inter-modal deep semantic collaborative modeling under multiple attention perspectives, and complete local model updating through a local optimizer, and then embed or upload the local features to the central server.

[0007] Step S4, introduce a federated attention aggregation module, which aggregates the uploaded features by adaptive attention weights and dynamically adjusts the contribution of features of different modalities.

[0008] Step S5, through multiple rounds of iterative updating, the federated multi-modal attention learning framework shares high-quality multi-modal models among multiple clients under the premise of protecting data privacy.

[0009] Further, in the step S1, the modal feature encoding step of the multi-modal feature encoding module is as follows: In the multi-modal federated learning-based text-image semantic modeling method, the input data of the client is composed of an image modality I and a text modality T; Let the input image modality feature be , and the text modality feature be , respectively obtain query vectors, key vectors, and value vectors through linear projection: wherein, indicates that the image modality I is linearly transformed by the projection matrix to obtain the query vector of the image modality. indicates that the text modality feature T is linearly transformed by the projection matrix to obtain the key vector of the text modality. indicates that the text modality feature T is linearly transformed by the projection matrix to obtain the value vector of the text modality. is a learnable projection matrix. In this way, the basic representation of inter-modal semantic alignment is realized.

[0010] Further, the local cross-modal self-attention modeling module is used to realize deep interaction between image modal and text modal, and a self-attention mechanism is adopted to construct a semantic alignment representation. First, the attention weight is calculated: wherein, to calculate the similarity between the query vector and the key vector; is a scaling factor; is to convert the similarity into an attention weight; is to weight the value vector according to the attention weight, and obtain a fused feature representation; The representation form of cross-modal interaction is: wherein, is a cross-modal fused feature; is an image feature, and is used as a query vector to query semantic information in the text; is a key vector; This operation represents that the image feature in the query space is used to weight and fuse the text semantic information.

[0011] Further, the local cross-modal self-attention modeling module uses a multi-head attention mechanism to calculate the representations of multiple subspaces in parallel: wherein h is the number of heads, is the projection parameter of the i-th head.

[0012] Further, the local cross-modal self-attention modeling module introduces residual connection and layer normalization to enhance training stability and deep modeling capability: Further, the semantic abstraction capability is enhanced through a feedforward neural network mapping: wherein FFN is a feedforward network, usually including two fully connected layers and an activation function.

[0013] Further, the aggregation step of the federated attention aggregation module is as follows: Suppose there are K clients participating in federated training, and the local representation uploaded by each client is The server first uses an attention query vector Local representation of each client Matching, calculating attention weight: Wherein W is a learnable linear transformation parameter, used for uniform mapping of local representation; The importance weight of the kth client feature is represented.Finally, the server aggregates all client features according to the attention weight to obtain the global fusion representation Wherein Will be used to guide the next round of model broadcast or as an auxiliary supervision signal, guide the global consistency alignment of the model to the cross-modal information.

[0014] Further, the optimization step of the multi-head collaborative fusion optimization module is as follows: There are two kinds of modal embedding representations, image modal is , and the text modal feature is Wherein n is the sequence length, and d is the feature dimension; Then send the two kinds of modal into different attention subspaces to construct multi-head fusion representation: For the hth fusion head, the cross-modal attention calculation between image and text is carried out respectively: Obtain the text fusion representation under the guidance of the image and the image fusion representation under the guidance of the text: Concatenate all multi-head fusion results and map them back to the original feature dimension: The importance of different fusion components is dynamically controlled by introducing a gating mechanism, thereby improving the collaborative expression effect: Wherein Indicates Activation function, and indicates element-wise multiplication, .

[0015] The present application improves a multi-modal federated learning method based on self-attention is provided herein, compared with the prior art, has the following improvements and advantages: ​The application improves the model performance and generalization ability of multi-modal tasks in a distributed environment, and compared with existing federated multi-modal learning methods, introduces a cross-modal alignment attention module and a local modal attention enhancement mechanism to improve the federated aggregation effect from two dimensions of semantic alignment and inter-modal structure modeling. Without relying on privacy enhancement mechanisms, FedMMAtt can efficiently adapt to real distribution differences and improve the global performance of image-text matching tasks.

[0016] The application designs a local modal attention and cross-modal alignment mechanism based on the Transformer structure, realizes efficient fusion of heterogeneous client features, constructs a dual-modal attention alignment module to guide the semantic complementarity between client local image-text features, and significantly improves the image-text matching accuracy. Large-scale experiments are carried out on MS-COCO and Flickr30K datasets, and the results show that the federated multi-modal attention learning framework is superior to existing federated multi-modal learning methods in various indicators, verifying its effectiveness and universality. BRIEF DESCRIPTION OF DRAWINGS

[0017] The application will be further explained below in combination with the drawings and embodiments: Figure 1 The figure is a schematic diagram of the system framework of the application; Figure 2 The figure is a schematic diagram of the self-attention cross-modal modeling module of the application; Figure 3 The figure is a schematic diagram of the federated attention aggregation module of the application; Figure 4 The figure is a schematic diagram of the multi-head collaborative fusion optimization module of the application; Figure 5 The figure is a schematic diagram of the accuracy trend of different methods of the application in 0-200 rounds of communication; Figure 6 The figure is a schematic diagram of the accuracy change of different methods of the application under different noise intensities; Figure 7 The figure is a schematic diagram of the client T-SNE visualization of the application under ordinary model training; Figure 8 The figure is a schematic diagram of the client T-SNE visualization of the application under FedMMAtt training; Figure 9 The figure is a schematic diagram of different experiments of the application under different alpha of dirichlet distribution to control the reservation module; Figure 10 The figure is a schematic diagram of the performance of the self-attention based multi-modal federated learning model of the application compared with the current better method CreamFL. DETAILED DESCRIPTION

[0018] The application will be further explained below in combination with the drawings and embodiments:Figure 1 The present invention has been described in detail below, and the technical solutions in the embodiments of the present invention have been clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 10 As shown, this invention provides an improved multimodal federated learning method based on self-attention, comprising the following steps: Step S1: Design a multimodal feature encoding module on the client side, and then use an image encoder and a text encoder to represent the image modality and text modality data respectively. Both are based on the Transformer backbone structure with self-attention mechanism to capture the deep feature semantics inside each modality. Step S2: Introduce a local cross-modal self-attention modeling module. For each client, conduct multiple rounds of local training, utilize intra-modal self-attention and inter-modal cross-attention to extract multi-level, high-semantic local multimodal features, and then model fine-grained semantic alignment between images and text through a bidirectional cross-attention mechanism. Step S3: Combine the multi-head collaborative fusion optimization module, the locally uploaded multimodal features are sent to multiple federated attention heads to construct a multi-view global representation. Deep semantic collaborative modeling between modalities is performed under the multi-attention perspective, and the local model is updated through the local optimizer. Then, the local features are embedded or the gradient is uploaded to the central server.

[0020] Step S4 introduces a federated attention aggregation module, which weights and aggregates uploaded features using adaptive attention weights and dynamically adjusts the feature contributions of different modalities. The central server acts as the global coordinator, responsible for information fusion among multiple clients. Unlike traditional federated averaging, this invention introduces a federated attention aggregation module. This module weights and aggregates uploaded features using adaptive attention weights, considering not only the heterogeneity of data among clients but also dynamically adjusting the feature contributions of different modalities, effectively avoiding overfitting or neglecting a particular modality during the aggregation process.

[0021] Step S5: Through multiple rounds of iterative updates, the federated multimodal attention learning framework enables multiple clients to share a high-quality multimodal model while protecting data privacy. Based on self-attention, the multimodal federated learning framework can achieve the sharing of a high-quality multimodal model among multiple clients while protecting data privacy, and is widely applicable to multimodal tasks such as image and text retrieval, cross-modal matching, and intelligent question answering.

[0022] like Figure 2As shown, multimodal feature encoding and self-attention cross-modal modeling MACM: (1) Modal feature encoding: In the multimodal federated learning-based image-text semantic modeling method, the input data of the client is composed of an image modality I and a text modality T; Let the input image modality feature be , and the text modality feature be , respectively through linear projection to obtain query vector, key vector, value vector: Among them, indicates that the image modality I is linearly transformed through the projection matrix to obtain the query vector of the image modality ; indicates that the text modality feature T is linearly transformed through the projection matrix to obtain the key vector of the text modality ; indicates that the text modality feature T is linearly transformed through the projection matrix to obtain the value vector of the text modality ; is a learnable projection matrix. Through this way, the basic representation of semantic alignment between modalities is realized.

[0023] (2) Self-attention cross-modal modeling: To realize the deep interaction between the image modality and the text modality, the application adopts a self-attention mechanism to construct a semantic alignment representation.

[0024] First, calculate the attention weight: Among them, is the similarity between the query vector and the key vector; is a scaling factor; is the similarity converted into an attention weight; is the weighted sum of the value vector according to the attention weight, to obtain the fused feature representation; The representation form of cross-modal interaction is: Among them, is the cross-modal fused feature; is the image feature and is used as the query vector to query the semantic information in the text; is the query vector. The operation represents that the image features are weighted and fused in the query space according to the text semantics.

[0025] (3) Multi-head attention mechanism extension To improve the feature expression ability, a multi-head attention mechanism is used to calculate multiple subspaces in parallel: where h is the number of heads, is the projection parameter of the i-th head.

[0026] (4) Cross-modal residual fusion and normalization In order to enhance the training stability and deep modeling ability, residual connection and layer normalization are introduced: Further through the feedforward neural network mapping to enhance the semantic abstraction ability: where FFN is a feedforward network, usually containing two fully connected layers and an activation function.

[0027] This module effectively realizes the deep interaction and alignment between different modalities through linear transformation, cross-modal attention, multi-head mechanism and normalization mechanism, and provides rich fusion feature representation for downstream tasks.

[0028] In the multi-modal federated learning framework, the features extracted by the clients have significant differences due to the differences in heterogeneous modalities and local data distribution. To achieve more fine-grained personalized feature fusion and global model update, the present invention introduces a federated attention aggregation module. The module introduces an attention mechanism on the server to dynamically assign different aggregation weights to the features uploaded by each client, thereby improving the robustness and personalized expression ability of the federated model in multi-modal tasks.

[0029] The federated attention aggregation module (FFA) is as shown in Figure 3 , assuming that there are K clients participating in federated training, and the local representation uploaded by each client is The server first matches the attention query vector with the local representation of each client to calculate the attention weight: where W is a learnable linear transformation parameter for uniformly mapping the local representation; The importance weight of the kth client feature. Finally, the server aggregates all the client features according to the attention weight to obtain the global fusion representation : wherein will be used to guide the next round of model broadcast or as an auxiliary supervision signal to guide the global consistency alignment of the model to the cross-modal information. This module not only considers the significant differences in feature expression between clients, but also has good scalability and can adapt to situations such as modal imbalance and modal weight changes in a multi-modal scenario.

[0030] (4) Multi-head collaborative fusion optimization module The multi-head collaborative fusion optimization module is shown in Figure 4 , first, the embedding representation of two modalities is set as , and the text modal feature is , wherein n is the sequence length and d is the feature dimension. The two modalities are sent into different attention subspaces to construct multi-head fusion representation: For the hth fusion head, the cross-modal attention calculation between the image and text modalities is performed respectively: The text fusion representation under the guidance of the image and the image fusion representation under the guidance of the text are obtained: All multi-head fusion results are spliced and mapped back to the original feature dimension: The gating mechanism is introduced to dynamically control the importance of different fusion components, thereby improving the collaborative expression effect: wherein is an activation function, and is an element-wise multiplication. .

[0031] Through the above module design, on the basis of the multi-head attention mechanism, cross-modal interaction semantic modeling and gating control mechanism are introduced, which effectively alleviates the conflict problem between modalities and realizes fine-grained and multi-perspective collaborative optimization. Experimental results show that this module has achieved significant performance improvement in multi-modal classification, retrieval and federated tasks.

[0032] The multi-modal federated learning algorithm based on self-attention performs as follows: Embodiment 1: The proposed self-attention multi-modal federated learning model (FedMMAtt) is experimentally evaluated on two standard large-scale multi-modal datasets. Meanwhile, to verify the effectiveness of the introduced modules, ablation experiments are also designed and conducted. The ablation experiments are only conducted on the MS-COCO dataset.

[0033] (1) Datasets MS-COCO Dataset: MS-COCO (Microsoft Common Objects in Context) is one of the most widely used large-scale datasets in the current mainstream multi-modal image-text matching and image-text retrieval research. The dataset contains over 120,000 images, each with 5 human-labeled natural language descriptions, covering 80 common object categories and various complex scenes. In this experiment, the Karpathy division standard is used: 113,287 images are used for training, 5,000 for validation, and 5,000 for testing. To simulate the non-IID environment in federated learning, the invention divides the data according to image categories and text semantics, constructing 10 heterogeneous clients, each with incomplete category data.

[0034] Flickr30K Dataset: Flickr30K is another commonly used dataset for image-text retrieval and multi-modal learning tasks, containing 31,783 images, each with 5 English descriptions, covering a large number of human behaviors and object interactions in real-world scenarios. The invention uses its official recommended division, with 29,783 images for training, 1,000 for validation, and 1,000 for testing. Similar to MS-COCO, the invention divides the training set according to image labels and description content into several clients, each with incomplete modal and category information, to simulate real-world cross-domain distribution and modal heterogeneity.

[0035] The experiment is conducted on a cloud server equipped with NVIDIA GeForce RTX 4090 GPU. The invention uses the current superior multi-modal Transformer architecture as the baseline and proposes a multi-modal federated learning method FedMMAtt based on self-attention mechanism. The model uses ViT-B / 16 as the image encoder and BERT-Base as the text encoder, and realizes deep-level fusion of image-text information through a multi-modal cross-attention module. On the server side, a global attention aggregation strategy is used to more effectively integrate the local knowledge uploaded by each client.

[0036] In terms of training strategy, the stochastic gradient descent (SGD) optimizer is used with a momentum setting of 0.9 and a weight decay of 1e-4. The initial learning rate is 0.0001, and a linear warm-up strategy is used to smoothly increase the learning rate from 0 to the target value over the first 5 rounds; after the 6th round, a cosine annealing schedule is used to dynamically reduce the learning rate. The local epoch for each round of federal training is set to 5, the batch size is 32, and a total of 200 rounds of communication are performed.

[0037] In terms of loss function, the cross-entropy loss and contrastive learning loss are combined to optimize both classification accuracy and modality alignment capability. In the model evaluation stage, image-text retrieval experiments are conducted on the MS-COCO and Flickr30K datasets, and the evaluation metrics include image-text matching Top-k retrieval accuracy (R@1, R@5, R@10) and average precision mean (mAP).

[0038] (2) Model performance comparison (21) Accuracy comparison analysis of each method under the number of communication rounds To comprehensively evaluate the performance of the proposed self-attention-based multi-modal federated learning model (FedMMAtt), the invention conducts a federal image-text matching experiment on the MS-COCO dataset and compares the model accuracy of various mainstream federated learning algorithms under 200 rounds of communication. The comparison methods include: FedAvg, FedProx, FedBN, MOON, CreamFL and the proposed method FedMMAtt. During the experiment, all models maintain the same initial parameter settings and optimizer configurations to ensure fairness in comparison.

[0039] From Figure 5As can be seen: FedAvg and FedProx perform slowly in the initial communication stage, and the model accuracy is maintained below 10% in the first 100 rounds, and only slowly increases in the later stage, eventually reaching about 52.1% and 53.7% respectively, indicating that the classic federated average strategy is not suitable for non-IID under the multi-modal task. FedBN improves the individualization ability through the separation of the local BN layer, and compared with FedAvg, the convergence speed is still slow, and the final accuracy is 50.9%, which still has the problem of semantic expression deviation between modes. MOON introduces a contrast learning mechanism to guide the client to maintain consistent semantic representation, and shows a faster convergence trend in the later communication, with an accuracy of 58.6%. CreamFL introduces an inter-module intervention mechanism in the fusion stage, effectively improving the global alignment effect, and the accuracy is stably improved to 63.1%, surpassing the aforementioned baseline methods in most rounds. FedMMAtt performs most significantly, breaking through 30% around the 60th round, much earlier than other methods, indicating that it has stronger early generalization ability. In the subsequent communication process, its accuracy rapidly rises and stabilizes after the 120th round, reaching a highest level of 68.9%, significantly better than all comparison methods, verifying the effectiveness of the multi-modal feature interaction modeling and federated attention aggregation mechanism proposed by the present application.

[0040] In addition, it can be observed that FedMMAtt not only leads in the precision index, but also has faster convergence speed and stronger training stability, indicating that it has good robustness and generalization ability in the non-IID multi-modal federated learning scene.

[0041] (22) Robustness evaluation after adding noise In the multi-modal federated learning scene, due to the heterogeneity of client devices, complex acquisition environment and unstable transmission process, the model training will be disturbed by different degrees of noise, especially image blur, text interference, feature drift and other problems. Therefore, in order to further verify the robustness of the proposed model FedMMAtt, the present application designs a group of noise interference experiments to evaluate the accuracy change trend of each method under different noise intensity.

[0042] The present application simulates the disturbed data in actual deployment by adding Gaussian noise to the image modal feature, and sets multiple noise levels (0, 5, 10, 15, 20) to represent different severity of interference intensity. The communication rounds and other experimental settings are kept the same, and the performance of FedAvg, FedBN, FedProx, MOON, CreamFL and the proposed FedMMAtt under each noise level is compared and analyzed.

[0043] From Figure 6In the middle, the accuracy of all methods decreases with the increase of noise intensity, indicating that noise has a significant impact on the performance of the multi-modal federated model; FedAvg and FedBN have the most obvious accuracy drop when the noise intensity is high (15 and above), indicating that their model generalization ability and feature noise suppression ability are weak; FedProx and MOON are relatively stable, partly due to the introduction of personalized regularization terms and consistency constraints in local training; CreamFL still maintains good precision performance under moderate noise levels with the help of its modal fusion strategy, showing a certain degree of robustness; FedMMAtt achieves the best performance at each noise level, with an accuracy of 65.7% (no noise) smoothly decreasing to 33.8% (strongest noise), with the smallest overall decline, showing strong anti-noise ability.

[0044] This result fully verifies the effectiveness of the multi-modal feature interaction modeling mechanism and federated attention fusion mechanism proposed in the present application in the face of feature disturbance. In particular, the multi-head collaborative fusion optimization module (MCFOM) can assign different weights to different modal sub-features in multiple attention subspaces, allowing the model to extract key semantics even in the presence of partial information damage, thereby improving overall robustness.

[0045] (23) Comparative analysis of image-text retrieval performance To further verify the effectiveness of the FedMMAtt model proposed in the present application in multi-modal understanding tasks, the present application conducts in-depth analysis of the model performance on the image-text retrieval task. Two mainstream multi-modal data sets, MS-COCO and Flickr30K, are selected for evaluation, and image retrieval text (i2t) and text retrieval image (t2i) are used in two directions, with Recall@K (K=1, 5, 10) as the evaluation indicator, specifically i2t_R@1, i2t_R@5, i2t_R@10 and t2i_R@1, t2i_R@5, t2i_R@10, R@1_sum represents the sum of i2t_R@1 and t2i_R@1, used to measure the performance of the model in both directions.

[0046] Table 1 Performance comparison of different federated methods on MS-COCO image-text retrieval task (1K test set) (unit: %) Table 2 Performance comparison of different methods on Flickr30K image-text retrieval task (1K test set) (unit: %) From Table 1 (MS-COCO), it can be observed that FedMMAtt achieves the optimal performance on all indicators, especially leading other methods by a large margin on the R@1 indicator, with i2t_R@1 reaching 52.6%, t2i_R@1 reaching 39.6%, and comprehensive R@1_sum reaching up to 92.2%, which is about 7.1 percentage points higher than the second-best CreamFL. This indicates that FedMMAtt has a significant advantage in high-precision retrieval and can better align the deep semantics between image and text modalities.

[0047] Table 2 shows the experimental results on the Flickr30K dataset, and FedMMAtt also remains leading, with i2t_R@1 and t2i_R@1 reaching 50.7% and 38.5%, respectively, and R@1_sum reaching 89.2%. Although the sample size and diversity of Flickr30K are slightly lower than those of MS-COCO, FedMMAtt still exhibits strong generalization ability and robustness, and demonstrates better ability in modality alignment and semantic fusion.

[0048] Overall, FedMMAtt performs well on both datasets, significantly outperforming other comparative methods, verifying the effectiveness of the proposed modules (MACM, FAA, and MCFOM) in multi-modal feature modeling, federated alignment, and fusion optimization. Especially when facing cross-modal retrieval tasks, FedMMAtt provides more refined semantic modeling capabilities and more robust federated optimization strategies, which are suitable for practical multi-modal federated learning system deployment scenarios.

[0049] (24) Qualitative study of model drift To explore the model consistency and drift characteristics of clients under different federated training strategies, the invention uses the ordinary federated average method (VanillaFedAvg) and the improved FedMMAtt to train the model based on the same initial model, and uses the reduced feature representation for visualization analysis, as shown in Figure 7 and Figure 8 .

[0050] Figure 7 It shows that under the traditional federated average strategy, the feature distribution of each client presents a clear separation phenomenon: the red represented multimodal client (Multimodal client) feature cluster is concentrated but clearly separated from the image client (Image client1 & 2) distribution area, and there is a lack of shared feature space between the three types of clients. This indicates that in the ordinary federated method, the models trained by different data modalities have strong local feature bias, the feature representation space is not fully aligned, and there is a serious model drift phenomenon.

[0051] In contrast, Figure 8The client feature distribution under the FedMMAtt policy training is shown. It can be observed that the feature representations of the three types of clients are closer, and the distribution overlap between the multi-modal client and the image client is significantly enhanced. This shows that FedMMAtt can effectively alleviate the representation difference caused by heterogeneous data distribution, enhance the feature consistency between clients, thereby alleviating the model drift problem and improving the generalization ability and collaborative training effect of the model. The qualitative visualization result shows the effectiveness of FedMMAtt in the multi-source heterogeneous data environment, and it has a significant advantage over traditional methods in feature representation alignment, which helps to enhance the stability of the model in the federated system.

[0052] (25) Ablation experiment In order to analyze the specific contribution of each key module in FedMMAtt, the present application carries out a systematic ablation experiment. Figure 9 The performance of each model variant under different Non-IID data division settings (Dirichlet parameter β∈{0.1,0.3,0.5,0.7,1.0}) is shown. The present application constructs the following four configurations for comparison: -FAA-MCFOM: remove the federal attention aggregation module (FAA) and the multi-head collaborative fusion optimization module (MCFOM) of the server at the same time; +FAA-MCFOM: keep the FAA module and remove the MCFOM; -FAA+MCFOM: keep the MCFOM module and remove the FAA; FedMMAtt (complete): keep both the FAA and MCFOM modules.

[0053] From Figure 9 , it can be concluded that the -FAA-MCFOM variant performs the worst under all α settings, especially when α=0.1 (extremely heterogeneous), the accuracy is only 52.3%, indicating that the lack of FAA and MCFOM support in a highly heterogeneous scenario will severely weaken the model's ability to adapt to non-independent data.

[0054] MCFOM is more critical in high-heterogeneous scenarios: when α=0.1 and α=0.3, -FAA+MCFOM is significantly better than +FAA-MCFOM, indicating that the MCFOM module, through multi-head fusion and gating mechanisms, can more effectively alleviate the conflict and inconsistency between modalities in the case of extremely dispersed client data, significantly improving the model's robustness.

[0055] FAA further improves the aggregation efficiency under low heterogeneity: As alpha increases (e.g., alpha = 0.7 or alpha = 1.0), +FAA-MCFOM gradually approaches and even slightly surpasses -FAA+MCFOM, indicating that FAA can more fully exert its advantages in feature aggregation in an environment where data is evenly distributed among clients.

[0056] The complete model performance is always optimal: FedMMAtt achieves the highest accuracy in all alpha settings, especially when alpha is 0.5 or below, significantly outperforming all comparative variants. For example, when alpha is 0.1, its accuracy is 64.0%, which is more than 11.7% higher than the base version. This verifies that the FAA and MCFOM modules have good complementarity and synergy in federated heterogeneous multi-modal tasks.

[0057] In summary, the performance improvement of FedMMAtt is not due to a single module, but rather the combined effect of the global-aware aggregation capability of FAA and the modal-level adaptive coordination mechanism of MCFOM. In particular, in a highly heterogeneous data environment, the coordination mechanism is particularly critical for the robustness of the model.

[0058] (26) Effect of different sampling frequencies on model stability In the process of federated learning, the sampling rate of clients (i.e., the proportion of clients selected for training in each round of communication) has an important impact on model performance and stability. To evaluate the generalization ability and robustness of FedMMAtt under different sampling rates, the inventors further designed a comparative experiment as shown in Figure 7 , gradually increasing the proportion of participating clients in the range of 0.05 to 0.25 with a step size of 0.025, and recording the final accuracy trend of the model on the test set.

[0059] As shown in Figure 10 , the following observations can be made: The accuracy gradually increases as the sampling rate increases: FedMMAtt and CreamFL both show a performance increasing trend under different sampling rates, indicating that higher client participation can effectively reduce local bias and improve the consistency and generalization ability of the global model.

[0060] FedMMAtt outperforms the comparative methods at each sampling rate: At the lowest sampling rate (0.05), the accuracy of FedMMAtt is 63.2%, which is ahead of CreamFL's 61.2%; as the sampling rate increases to 0.25, FedMMAtt reaches 65.4%, stably leading CreamFL by at least 3 percentage points, verifying that the method still has strong representation ability under limited communication budget.

[0061] Model stability performs better: The error band corresponding to the FedMMAtt curve is significantly narrower than CreamFL, especially in the low sampling rate segment (0.05-0.15), showing lower performance fluctuations. This indicates that the FAA and MCFOM modules introduced by FedMMAtt can still effectively aggregate high-quality semantic information in the case of insufficient client participation, alleviating the model instability problem caused by insufficient participation samples.

[0062] In summary, FedMMAtt exhibits superior performance and strong stability under different sampling rates, especially in real-world scenarios where resources are limited or client response rates are unstable, making it more practical and adaptable for deployment.

Claims

1. A self-attention-based multi-modal federated learning method, characterized in that, Comprise the following steps: Step S1, design a multi-modal feature encoding module on the client, then represent the image modal and text modal data through image encoder and text encoder respectively, both based on the Transformer backbone structure of self-attention mechanism, capture the deep feature semantics inside each modal; Step S2, introduce a local cross-modal self-attention modeling module, through multiple rounds of local training on each client, extract multi-level and high semantic local multi-modal features by using intra-modal self-attention and cross-modal attention, and then model the fine-grained semantic alignment between images and texts through a bidirectional cross-attention mechanism; Step S3, combine a multi-head collaborative fusion optimization module, the locally uploaded multi-modal features are sent into multiple federated attention heads, a multi-perspective global representation is constructed, deep semantic collaborative modeling between modalities is performed under multiple attention perspectives, and local model updating is completed through a local optimizer, then the local features are embedded or uploaded to the central server as gradients; Step S4, introduce a federated attention aggregation module, weight and aggregate the uploaded features through adaptive attention weights, and dynamically adjust the feature contribution of different modalities; Step S5, through multiple rounds of iterative updating, the federated multi-modal attention learning framework shares high-quality multi-modal models among multiple clients under the premise of protecting data privacy.

2. The self-attention-based multi-modal federated learning method according to claim 1, wherein, In step S1, the modal feature encoding step of the multi-modal feature encoding module is as follows: In the image-text semantic modeling method based on multi-modal federated learning, the input data of the client is composed of image modal I and text modal T; Let the input image modality feature be , and the text modality feature be , respectively obtain the query vector, the key vector, and the value vector through linear projection: wherein, represents a linear transformation of the image modality I by a projection matrix to obtain a query vector of the image modality ; represents that the text modality feature T is linearly transformed by a projection matrix to obtain the key vector of the text modality ; represents a linear transformation of the text modality features T by a projection matrix to obtain a value vector for the text modality ; The projection matrix is learnable. In this way, the basic representation of inter-modal semantic alignment is realized.

3. The self-attention-based multi-modal federated learning method according to claim 1, wherein, The local cross-modal self-attention modeling module is used to realize deep interaction between image modal and text modal, and adopts a self-attention mechanism to construct a semantic alignment representation; First, calculate the attention weight: wherein, to compute the similarity between the query vector and the key vector; is a scaling factor; to convert the similarity into attention weights; to obtain a fused feature representation by weighted sum of the value vectors according to the attention weights; The representation form of cross-modal interaction is: wherein, is a feature for cross-modal fusion; Image features are used as query vectors to query semantic information in text. is a healthy vector; This operation represents that the image feature performs weighted fusion on the text semantics in the query space.

4. The self-attention-based multi-modal federated learning method according to claim 3, characterized in that, The local cross-modal self-attention modeling module uses a multi-head attention mechanism to perform parallel calculation on the representations of multiple subspaces: where h is the number of heads, is the projection parameter for the i-th head.

5. The self-attention-based multi-modal federated learning method according to claim 1, wherein, In order to enhance the training stability and deep modeling capability, the local cross-modal self-attention modeling module introduces residual connection and layer normalization: Further enhance the semantic abstraction capability through a feedforward neural network mapping: Where FFN is a feedforward network, usually containing two fully connected layers and an activation function.

6. The self-attention-based multi-modal federated learning method according to claim 1, wherein, The aggregation step of the federated attention aggregation module is as follows: Let K clients participate in federated training, and the local representation uploaded by each client is The server first matches the attention query vector with the local representation of each client to calculate the attention weight: where W is a learnable linear transformation parameter for uniform mapping of local representations; represents the importance weight of the kth client feature. Finally, the server aggregates all client features according to the attention weight to obtain the global fusion representation : wherein will be used to guide the next round of model broadcast or as an auxiliary supervision signal to guide the global consistency alignment of the model to cross-modal information.

7. The self-attention-based multi-modal federated learning method according to claim 1, wherein, The optimization step of the multi-head collaborative fusion optimization module is as follows: The embedded representation with two modalities is respectively , the text modality feature is where n is the sequence length and d is the feature dimension; Then, the two modalities are sent into different attention subspaces to construct multi-head fusion representations: For the hth fusion head, the cross-attention calculation between image modal and text modal is performed respectively: Obtain the text fusion representation guided by the image and the image fusion representation guided by the text: Concatenate all multi-head fusion results and map them back to the original feature dimension: Introduce a gating mechanism to dynamically control the importance of different fusion components, thereby improving the collaborative expression effect: wherein represents an activation function, and represents element-wise multiplication, .

Images