Federal multi-modal learning method based on alignment enhancement fusion
By employing an alignment-enhanced fusion approach in a federated learning environment and leveraging collaborative training between the client and the cloud, the privacy protection and data heterogeneity issues of multimodal data in a distributed environment are addressed. This approach enables efficient multimodal feature fusion and classification, improving learning effectiveness and resource utilization efficiency.
Patent Information
- Application Number
- CN202510894588.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-30
- Publication Date
- 2025-10-17
AI Technical Summary
Existing multimodal learning solutions suffer from insufficient privacy protection, limited ability to handle data heterogeneity, lack of edge-cloud collaborative optimization, poor adaptability to scenarios with scarce labels, and low communication efficiency in distributed environments. In particular, they are difficult to effectively utilize unlabeled multimodal data across clients for collaborative learning in federated learning scenarios.
A federated multimodal learning method based on alignment-enhanced fusion is adopted. Through the end-cloud collaborative architecture, the client feature encoder is used to perform feature extraction and dimension alignment without uploading the original local data, generating a pseudo-label set. The training is then carried out in stages on the client and the cloud, and finally the global feature encoder parameters are aggregated on the cloud to achieve effective fusion and classification of multimodal features.
While protecting privacy, it improves learning effectiveness, solves the problems of data heterogeneity and label scarcity, makes full use of edge cloud resources, reduces communication overhead, and significantly improves the performance of multimodal feature fusion.
Smart Images
Figure CN120806064A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, Internet of Things and edge computing, and particularly relates to a federated multi-modal learning method based on alignment enhancement fusion. BACKGROUND
[0002] With the rapid development of information technology and the popularity of intelligent devices, the current stage is in an era of data explosion. Among them, multi-modal data composed of various sources and forms of data such as images, texts, speeches, and videos has become the main carrier of information. How to effectively utilize these massive, unlabeled multi-modal data to learn valuable knowledge and representations from them has become one of the core challenges in the field of artificial intelligence, especially in the field of machine learning. To meet this challenge, a number of key technologies have emerged.
[0003] First, the Multimodal Self-Supervised Learning technology provides an effective solution. The core idea of this technology is to use the inherent correlation of the data itself as a supervision signal, for example, a picture and its corresponding text description are naturally a pair of positive samples. By designing specific pretext tasks such as Contrastive Learning or Cross-modal Consistency Learning, the model can learn the consistency and difference between different modal data without relying on human-labeled data, thereby obtaining high-quality data representations. This method greatly reduces the dependence on expensive human-labeled data.
[0004] Secondly, in practical applications, data is often scattered in different user terminal devices or institutions. Due to privacy protection, data security and regulatory compliance (such as GDPR, data security law), it becomes increasingly impractical to collect raw data to a central server for centralized training. For this reason, Federated Learning technology has emerged. Federated Learning is a distributed machine learning paradigm that allows multiple participants (clients) to train models locally using their own data without uploading raw data. Each participant only sends encrypted or obfuscated model update parameters (such as gradients, model weights) to the central server for aggregation, and the server then distributes the aggregated global model to each participant. In this way, federated learning realizes "data does not move, model moves", while protecting user privacy and data sovereignty, it collaboratively builds a global model with better performance.
[0005] In addition, in order to balance the computing and storage capabilities of terminal devices and the powerful computing power of cloud servers, an edge-cloud collaborative computing architecture has been widely applied. This architecture reasonably allocates computing tasks between terminals (edge side) and cloud servers (cloud side), processes some tasks with low computing intensity or high delay requirements on the edge side, and transfers complex tasks requiring massive computing resources to the cloud side for completion. This collaborative approach can fully utilize the computing resources of the entire network, reduce cloud load, and improve the overall operation efficiency and response speed of the system.
[0006] Under the above technical background, researchers have begun to explore how to perform multi-modal learning in a distributed scenario. Currently, some solutions for multi-modal data analysis have been proposed, and the technical solutions closest to the present application mainly focus on feature extraction and alignment of multi-modal data. However, they are usually performed on the premise that data is already centralized.
[0007] For example, in a technical solution named “Contrastive Multimodal Clustering” (CMC) proposed by Tian et al., the core is to align features of different modalities through contrastive learning. This method is mainly applied to a single computing node, assuming that multi-modal data (such as images and text) can be directly accessed by the node. The working principle is as follows: for the same data sample (such as a picture of a cat and its description “this is a cat”), the method will “pull” the representations of different modalities of the sample closer in the feature space, i.e., maximize the mutual information or similarity between them; at the same time, for different data samples (such as a picture of a cat and a picture of a dog), the representations of the samples are “pushed” away in the feature space, i.e., the similarity between them is minimized. Through this contrastive learning, the model can learn cross-modal and semantically consistent feature representations. However, the original intention of the CMC method is to handle centralized data, and it does not consider the federated learning scenario where data is distributed among multiple clients and cannot be shared.
[0008] For example, another FOCAL method proposed by Liu et al. proposes a technical solution for extracting shared information and modality-specific information from multi-modal data using decomposed orthogonal latent spaces. This method decomposes the feature representation of each modality into two parts through a sophisticated matrix decomposition technique: one part is the "shared component" common to all modalities, and the other part is the "private component" unique to the modality. In this way, the model can more finely deconstruct multi-modal data. However, this solution is also designed on the premise that data can be centrally accessed, and it requires a decomposition operation on the overall data matrix of all samples, which cannot be directly implemented in a federated learning framework with data isolation, and it also fails to solve the problem of coordinating multiple client data under the premise of protecting privacy.
[0009] However, these methods are all centralized learning solutions that require all multi-modal data to be centralized on a single node for processing. In a distributed environment, due to the limited amount of data from a single client and the problem of data heterogeneity, the performance of these methods is significantly reduced. More importantly, these methods require the upload of original multi-modal data to a central server, posing a serious risk of privacy leakage.
[0010] In summary, the current multi-modal learning solutions have the following problems:
[0011] 1. Insufficient privacy protection: Although existing federated multi-modal learning solutions do not directly share raw data, they still require the upload of model weight parameters. These parameters may leak raw data information through gradient inversion attacks and other methods, especially in the multi-modal scenario, attackers can further improve the success rate of attacks by exploiting the correlation between modalities.
[0012] 2. Limited ability to handle data heterogeneity: Existing solutions mainly handle data heterogeneity between clients through simple weight averaging or adaptive weight adjustment, but in the multi-modal scenario, different clients may have different modality combinations (such as some clients only having image data and some only having text data), and existing solutions are difficult to effectively handle this modality heterogeneity.
[0013] 3. Lack of end-to-cloud collaborative optimization: Existing solutions assign all computing tasks to either the client or the server, failing to fully utilize the advantages of end-to-cloud collaboration. The computing resources of the client are not fully utilized, and the server side also fails to leverage its powerful computing capabilities to assist multi-modal feature fusion.
[0014] 4. Poor adaptability to label-scarce scenarios: Existing solutions are mainly designed for scenarios with abundant labels, and in federated scenarios with limited labels, the performance is significantly reduced due to the inability to effectively utilize cross-client label-free multi-modal data for collaborative learning.
[0015] 5. Low communication efficiency: the existing scheme needs to frequently upload complete model weight parameters, and in a multi-modal scene, due to the large model size, the communication overhead is huge, which seriously affects the scalability of the system. SUMMARY
[0016] The present application aims to at least partially solve one of the technical problems in the related art.
[0017] To this end, the present application provides a federated multi-modal learning method based on alignment-enhanced fusion, which realizes efficient multi-modal federated learning without uploading original local data through an end-cloud collaborative architecture, and can greatly improve the privacy protection level and learning effect.
[0018] In order to achieve the above purpose, the present application adopts the following technical solutions:
[0019] The federated multi-modal learning method based on alignment-enhanced fusion provided by the present application comprises:
[0020] Step S100, using the feature encoders of each client participating in federated learning to perform feature extraction and dimension alignment on local multi-modal data to obtain projected features; the local multi-modal data is divided into local labeled multi-modal data and local unlabeled multi-modal data;
[0021] Step S200, generating fusion features based on the projected features of the local labeled multi-modal data and the local unlabeled multi-modal data of each client and constructing a pseudo-label set;
[0022] Step S300, performing local training in two stages at each client; wherein,
[0023] In the first stage, the feature encoders are pre-trained using self-supervised contrastive learning with local unlabeled multi-modal data;
[0024] In the second stage, the feature encoders are pre-trained using supervised contrastive learning with local labeled multi-modal data and a pseudo-label set to obtain trained local feature encoders;
[0025] Step S400, aggregating the local feature encoder parameters of each client at the cloud and updating the global feature encoder, and downloading the updated global feature encoder parameters to each client.
[0026] In some embodiments, there are K clients participating in federated learning in total, and for a client k among them, step S100 comprises:
[0027] Step S110, client k receives n k local multi-modal data samples, and let the received i local multi-modal data samples be wherein J represents the number of different modalities of data contained in the local multi-modal data sample of the client k, j represents the modality number of the data in the local multi-modal data sample of the client k;
[0028] Step S120, the feature encoder of the client k performs feature extraction on each modality of data :
[0029]
[0030] wherein, represents the local multi-modal data sample of the client k extracted by the feature encoder of the client k the features of the jth modality of data in the local multi-modal data sample of the client k; Flatten represents a flattening operation on a multi-dimensional matrix; f encj represents the feature encoder for the jth modality of data;
[0031] Step S130, mapping the features of different dimensions to a unified dimension d to obtain projected features : the projected features are obtained according to the following formula:
[0032]
[0033] wherein, represents a transformation operation of mapping the features of different dimensions to a unified dimension d; Norm represents a normalization operation.
[0034] In some embodiments, step S200 comprises:
[0035] Step S210, the client k performs P times of random fusion enhancement on the projected features of each local multi-modal data sample :
[0036]
[0037] wherein, is the feature after the pth fusion enhancement on the local multi-modal data sample ; α jp is a random weight coefficient, satisfying jp =1.
[0038] Step S220, constructing the feature matrix of the labeled data and the feature matrix of the unlabeled data
[0039]
[0040] wherein, indicates The Ljth modal data with local labels The corresponding n-th labeled fusion enhanced feature, express The Uith modality data in the local unlabeled The corresponding m-th unlabeled fusion enhanced feature;
[0041] Step S230: Calculate the cosine similarity between the unlabeled features and the labeled features For each unlabeled feature The labeled feature with the highest cosine similarity is selected as the candidate labeled feature, and the label corresponding to each candidate labeled feature whose cosine similarity exceeds the preset threshold is assigned to the local unlabeled multimodal data to form a pseudo label set D * .
[0042] In some embodiments, step S300 includes:
[0043] Step S310: In the first stage, the client pre-trains the local feature encoder using local unlabeled multimodal data and self-supervised contrastive learning;
[0044] Step S320, the second stage, using local labeled multimodal data and pseudo label set D * Perform supervised contrastive learning on the pre-trained local feature encoder and calculate the contrastive fusion loss L mix :
[0045]
[0046] Where i' is D * The sample number in ; D * The fusion feature of the sample numbered i' in the The index set of samples with the same label; Represents the fusion features of the sample with sequence number p' in P(i'); represents the fusion feature of the sample with sequence number s in P(i'); D * / P(i') is the * The set obtained by removing P(i'); a is D * / The sample number in P(i'); Indicates D * The sequence number in / P(i') is the fusion feature of sample a; τ represents the temperature coefficient.
[0047] In some embodiments, step S400 includes:
[0048] The cloud uses the trained local feature encoder parameters θ uploaded by each client. k Perform federated average aggregation to obtain the updated global feature encoder parameters θ global , the updated global feature encoder parameters are sent to each client; wherein, the updated global feature encoder parameters θ global According to the following formula:
[0049]
[0050] Among them, θ global is the updated global parameter; K is the number of clients participating in federated learning; n k is the number of local multimodal data samples contained in client k, and N is the total number of samples participating in federated training.
[0051] In some embodiments, after repeatedly executing steps S100 to S400 several times until reaching a set upper limit of the number of iterations, the federated multimodal learning method further includes: step S500, performing weighted enhanced contrast fusion in the cloud, including:
[0052] Step S510: The cloud receives the deep multimodal features (z1, ..., z2) extracted by the trained local feature encoder uploaded by each client. j ,…,z J ), thereby constructing a feature set D containing labels L and the unlabeled feature set D U The global feature set of
[0053] Step S520: for each deep multimodal feature z j , by fusing dynamic weight w(e) j and mass weight w(hp) j Get z j The weight W j , the dynamic weight w(e) j Calculated based on the attention mechanism, the quality weight w(hp) j Used to evaluate the clustering quality of unlabeled data;
[0054] Step S530: for the i-th local multimodal data sample Utilize the deep multimodal features of the sample and its corresponding weight W j Get the multimodal fusion feature v of the sample i ;
[0055] Step S540: Use the labeled features in the cloud to perform supervised contrastive learning. The loss function used is L surp :
[0056]
[0057] wherein D L is the set of all labeled fusion features; i" is the sample index in D L ; is the fusion feature of the sample with index i" in D L ; S(i") is the set of feature indices with the same label as D ; is the fusion feature of the sample with index s in S(i"); D L / S(i") is the set of samples in D L / S(i") after removing S(i"); a' is the sample index in D L / S(i"); is the fusion feature of the sample with index a' in D L / S(i"); τ is the temperature coefficient;
[0058] Step S550, updating the weight parameters through gradient backpropagation;
[0059] Step S560, calculating the multi-modal fusion features using the updated weights as the optimized multi-modal fusion features.
[0060] In some embodiments, the calculation formula of the weight W j is as follows:
[0061] W j = (1 - λ)w(e) j + λw(hp) j
[0062]
[0063] μ j = tanh(W·z j +b)
[0064]
[0065] wherein μ j is an intermediate variable; tanh() is an activation function; W is a neural network weight; b is a bias; λ is a hyperparameter; hp j is the Hopkins statistic coefficient of the jth modality.
[0066] In some embodiments, the federated multi-modal learning method further comprises step S600, performing multi-modal semi-supervised learning in the cloud, comprising:
[0067] Step S610, generating strong-weak data pairs required by semi-supervised learning based on the weight information obtained in step S500, for simulating the original-transformed data pair relationship in data augmentation;
[0068] Step S620, deploying and initializing the teacher model and the student model on the cloud, and training the teacher model and the student model simultaneously using the strong-weak data pairs to implement semi-supervised learning;
[0069] Step S630, updating the student model parameters based on consistency regularization, and updating the teacher model parameters based on the exponential mean index to maintain the consistency between the teacher model and the student model;
[0070] Step S640, the cloud end issuing the final teacher model parameters as global feature encoder parameters to each client.
[0071] In some embodiments, in step S610, based on the weight W j obtained in step S500, the strong data v strong and the weak data v weak are generated respectively according to the following formulas:
[0072]
[0073] wherein, Concat represents a connection operation; represents the jth deep multimodal feature of the ith local multimodal data sample; J is the number of different modal data contained in the ith local multimodal data sample.
[0074] In some embodiments, step S620 comprises:
[0075] Step S621, initializing the student model S_model and the teacher model T_model;
[0076] Step S622, forward propagation:
[0077] inputting the strong data v strong into the student model to obtain the predicted label y strong for the strong data v s , y s =S_model(v strong );
[0078] inputting the weak data v weak into the teacher model to obtain the predicted label y weak for the weak data v t , y t =T_model(v weak );
[0079] Step S623, total loss Ltotal Compute:
[0080] L toatal = L sup + lambda cons * L cons
[0081] L sup = CrossEntropy(y s , y true )
[0082] L cons = MSE(y s , y t )
[0083] Where L cons is a regular term function of MSE; L sup is the loss value obtained by supervised learning; CrossEntropy is a cross-entropy function; y true is the real sample label; lambda cons is a hyperparameter;
[0084] In step S630, the total loss L total Gradient is calculated based on consistency regularization, and the student model parameters are updated using back propagation; the teacher model parameters are updated by exponential moving average:
[0085]
[0086] Where alpha is the decay factor of the exponential average index; Indicates the model parameters obtained by the tth update of the teacher model, Indicates the model parameters obtained by the tth update of the student model.
[0087] Compared with the prior art, the present application has the following characteristics and beneficial effects:
[0088] The federated multi-modal learning method based on alignment enhancement fusion provided by the present application solves the problems of labeled data scarcity, data heterogeneity and privacy protection in multi-modal data fusion in a federated learning environment through training strategies at different stages. The technical scheme adopts a privacy-protected federated learning architecture, realizes effective fusion and classification of multi-modal features under the premise of ensuring that data does not leave the domain. BRIEF DESCRIPTION OF DRAWINGS
[0089] Figure 1 Fig. (a) and (b) are feature distribution obtained by centralized training and distributed training, respectively;
[0090] Figure 2is a federal multi-modal learning method flowchart based on alignment enhanced fusion provided by the embodiment of the application;
[0091] Figure 3 is Figure 2 the framework diagram of the specific implementation process of the method shown in the figure;
[0092] Figure 4 is the performance index comparison chart of the federal multi-modal learning method proposed by the embodiment of the application and the existing method on three standard multi-modal data sets. DETAILED DESCRIPTION
[0093] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be described clearly and completely below in combination with the drawings in the embodiment of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.
[0094] On the contrary, the present application covers any alternative, modification, equivalent method and scheme defined by the claims on the essence and scope of the present application. Further, in order to make the public better understand the present application, some specific details are described in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.
[0095] Referring to Figure 1 Through investigation, the applicant finds that multi-modal fusion in a distributed scenario has the characteristics of feature dispersion and poor clustering effect. The present application is based on this characteristic.
[0096] Referring to Figure 2 The embodiment of the present application provides a federal multi-modal learning method based on alignment enhanced fusion, comprising the following steps:
[0097] Step S100, using the feature encoder of each client participating in the federal learning to perform feature extraction and dimension alignment on the local multi-modal data, to obtain projected features; wherein the local multi-modal data is divided into local labeled multi-modal data and local unlabeled multi-modal data;
[0098] Step S200, generating fusion features based on the projected features of the local labeled multi-modal data and the local unlabeled multi-modal data of each client and constructing a pseudo label set;
[0099] Step S300, performing local training in two stages at each client; wherein,
[0100] In a first stage, a feature encoder is pre-trained by self-supervised contrastive learning using local unlabeled multi-modal data at the client side;
[0101] In a second stage, the pre-trained feature encoder is supervised contrastively learned using local labeled multi-modal data and a pseudo-label set at the client side, to obtain a trained local feature encoder;
[0102] In step S400, the local feature encoder parameters of each client are aggregated and the global feature encoder is updated at the cloud side, and the updated global feature encoder parameters are distributed to each client.
[0103] Referring to Figure 3 , a specific framework diagram of the federated multi-modal learning method is shown, and Figure 2 the specific implementation process of the federated multi-modal learning method provided by the first aspect of the present application is described in detail. Figure 3 The specific implementation process of the federated multi-modal learning method provided by the first aspect of the present application is described in detail.
[0104] In some embodiments, step S100 occurs in the initialization stage before the start of federated learning training. The feature encoder of each client extracts features and aligns dimensions from local multi-modal data, which runs in the local environment of each client and does not involve data transmission. Assuming that there are K clients participating in federated learning, for a client k, step S100 specifically includes:
[0105] In step S110, the client k receives n k local multi-modal data samples, and the i-th received local multi-modal data sample is denoted as where i represents the number of local multi-modal data samples, i = 1,..., n k , J represents the number of different modal data types in , and j represents the modal number of the data in ;
[0106] In step S120, the feature encoder of the client k extracts features from each type of modal data :
[0107]
[0108] wherein represents the feature of the j-th modal data in the local multi-modal data sample extracted by the feature encoder of the client k; Flatten represents a flattening operation on a multi-dimensional matrix; f encj represents a feature encoder for the j-th modal data, and each type of modal data is extracted by a corresponding feature encoder;
[0109] Step S130: Combine features of different dimensions Mapped to a unified dimension d, the standardized projection features are obtained Ensure that the projected features of all modes are located on the unit hypersphere, and the projected features According to the following formula:
[0110]
[0111] in, Represents the features of different dimensions The transformation operation mapped to the uniform dimension d; Norm represents the normalization operation of the current feature.
[0112] In some embodiments, step S200 occurs after feature alignment is completed and before contrastive learning training. Each client generates fused enhanced features based on the projected features and constructs pseudo labels. This is run in the local environment of each client and requires the use of local labeled and unlabeled data. Step S200 specifically includes:
[0113] Step S210: Client k processes each local multimodal data sample The projection features are randomly fused and enhanced P times:
[0114]
[0115] in, For local multimodal data samples The features after the p-th fusion enhancement; α jp is a random weight coefficient, satisfying ∑α jp =1;
[0116] Step S220: Construct a feature matrix with labeled data according to the following formula: and the feature matrix of unlabeled data
[0117]
[0118] in, express The Ljth modal data with local labels The corresponding n-th labeled fusion enhanced feature, express The Uith modality data in the local unlabeled The corresponding m-th unlabeled fusion enhanced feature;
[0119] Step S230: Calculate the cosine similarity between the unlabeled features and the labeled features For each unlabeled feature The labeled feature with the highest cosine similarity is selected as the candidate labeled feature, and the label corresponding to each candidate labeled feature whose cosine similarity exceeds the preset threshold ε is assigned to the local unlabeled multimodal data. The labels assigned to the local unlabeled multimodal data constitute a high-confidence pseudo-label set D * ; Calculated according to the following formula:
[0120]
[0121] Among them, sim() represents the calculation of cosine similarity.
[0122] In some embodiments, step S300 occurs after pseudo-label generation and before feature encoder parameter aggregation, and is run in a local training environment of the client. Step S300 specifically includes:
[0123] Step S310, first stage: self-supervised contrastive learning pre-training
[0124] The client uses a large amount of local unlabeled multimodal data to pre-train the feature encoder to establish a baseline model for the feature encoder and improve the quality of feature extraction;
[0125] Step S320, second stage: supervised contrastive learning fine-tuning
[0126] Using local labeled multimodal data and pseudo label set D * Perform supervised contrastive learning and calculate contrastive fusion loss L mix :
[0127]
[0128] Where i' is D * The sample number in ; D * The fusion feature of the sample numbered i' in the The index set of samples with the same label; Represents the fusion features of the sample with sequence number p' in P(i'); represents the fusion feature of the sample with sequence number s in P(i'); D * / P(i') is the * The set obtained by removing P(i'); a is D * / The sample number in P(i'); Indicates D * The sequence number in / P(i') is the fusion feature of sample a; τ represents the temperature coefficient.
[0129] Optionally, a regularization term related to the global feature encoder can also be introduced in the loss of local training to ensure the stability of the update of the local feature encoder and reduce parameter fluctuations.
[0130] In some embodiments, step S400 occurs after the local training of each client is completed, and the central server of the cloud aggregates the feature encoder parameters of each client and updates the global feature encoder parameters. Step S400 is run in a central server environment and only transmits the feature encoder parameters. Step S400 specifically includes:
[0131] The central server of the cloud aggregates the local feature encoder parameters θ k uploaded by each client according to the local feature encoder parameters θ global uploaded by each client according to the local feature encoder parameters θ global to obtain updated global feature encoder parameters θ
[0132]
[0133] wherein θ global is the updated global parameter; n k is the number of local multi-modal data samples contained by the client k, and N is the total number of samples participating in federated training.
[0134] It should be noted that when performing federated learning, steps S100-S400 are repeatedly executed several times until a set upper limit of iteration times is reached, and the federated learning is ended.
[0135] It can be understood that the above learning method provided by the embodiments of the present application protects the privacy data of each client and only shares at the model level, and can effectively solve the performance loss caused by insufficient data quantity of different clients.
[0136] In some embodiments, after the steps S100-S400 are repeatedly executed several times until a set upper limit of iteration times is reached, the embodiments of the present application further include step S500, weight enhancement contrast fusion performed in the cloud, specifically including:
[0137] Step S510, the cloud receives the deep multi-modal features (z j 1,…, z J extracted based on the trained local feature encoder uploaded by each client, thereby constructing a global feature set containing a labeled feature set D L and an unlabeled feature set D U .
[0138] Step S520, for each deep multi-modal feature zj , by fusing dynamic weight w(e) j and quality weight w(hp) j to obtain z j weight W j , the dynamic weight w(e) j is calculated based on the attention mechanism, and the quality weight w(hp) j is used to evaluate the clustering quality of the unlabeled data;
[0139] Further, the calculation formula of the weight W j is as follows:
[0140] W j = (1 - λ) w(e) j + λ w(hp) j
[0141]
[0142] μ j = tanh(W·z j +b)
[0143]
[0144] wherein μ j is an intermediate variable; tanh() is a commonly used activation function; W is a neural network weight; b is a bias; λ is a hyperparameter to adjust the importance factor of the two weights; hp j is a Hopkins statistic coefficient, and the method for obtaining the Hopkins statistic coefficient is an existing method.
[0145] Step S530, for the i-th local multi-modal data sample , the multi-modal fusion feature v i of the sample is obtained by using each deep multi-modal feature of the sample and the corresponding weight W j :
[0146]
[0147] wherein Concat represents a connection operation;
[0148] Step S540, supervised contrast learning is performed by using the labeled feature of the cloud, wherein the loss function L surp used is:
[0149]
[0150] wherein D L is a set of all labeled fusion features; i" is a set of all labeled fusion features; and i' is the index of the i-th local multi-modal data sample.L sample sequence number in D D L fusion feature of sample with sequence number i" in D feature index set with same label as S(i") fusion feature of sample with sequence number s in S(i") D L S(i") is a set obtained by removing S(i") from D L S(i") is a set obtained by removing S(i") from D L sample sequence number in D D L fusion feature of sample with sequence number a' in D τ is a temperature coefficient.
[0151] Step S550, updating the parameters of the neural network (i.e. the neural network for calculating the intermediate variable μ j ) through gradient backpropagation;
[0152] Step S560, calculating the multi-modal fusion feature by using the updated weights as the optimized multi-modal fusion feature for the classification task.
[0153] It can be understood that the embodiments of the present application only process the fusion feature based on the above step S500, without the need for the client to upload its own local data, which can achieve higher performance while protecting privacy.
[0154] In some embodiments, the embodiments of the present application further include step S600 of performing multi-modal semi-supervised learning on the cloud to solve the overfitting problem encountered by the global feature encoder during the training process after the above step S500. Step S600 specifically includes:
[0155] Step S610, generating strong and weak data pairs required for semi-supervised learning based on the weight information obtained in step S500, for simulating the original-transform data pair relationship in existing data augmentation;
[0156] Step S620, deploying and initializing the teacher model and the student model on the cloud, and training the teacher model and the student model simultaneously by using the generated strong and weak data pairs to realize semi-supervised learning;
[0157] Step S630, updating the parameters of the student model based on consistency regularization, and updating the parameters of the teacher model based on the exponential moving average (EMA) index to maintain the consistency of the teacher model and the student model;
[0158] Step S640, the cloud distributes the final teacher model parameters to each client as the global feature encoder parameters, and each client performs inference and classification on the local multi-modal data.
[0159] Further, in step S610, the weight W obtained in step S500 is used to generate the strong data v j strong and the weak data v weak according to the following formulas, respectively:
[0160]
[0161] Further, step S620 specifically includes:
[0162] Step S621, initialize the student model S_model and the teacher model T_model, both of which have the same network structure but the parameters are updated independently;
[0163] Step S622, forward propagation:
[0164] input the strong data v strong into the student model to obtain the predicted label y strong for the strong data v s , y s =S_model(v strong );
[0165] input the weak data v weak into the teacher model to obtain the predicted label y weak for the weak data v t , y t =T_model(v weak );
[0166] Step S623, total loss L total calculation:
[0167] L total =L sup +λ cons *L cons
[0168] L sup =CrossEntropy(y s ,y true )
[0169] L cons =MSE(y s ,y t )
[0170] wherein, L cons is a regular term function of MSE; L sup is a loss value obtained by supervised learning; CrossEntropy is a commonly used cross-entropy loss function; y true is the real label of the sample; λcons as a hyperparameter, controls the degree of influence on the result.
[0171] Further, the step S630 specifically comprises:
[0172] Step S631, student model parameter updating:
[0173] Calculate the total loss L based on consistency regularization total Gradient, update student model parameters using backpropagation
[0174] Step S632, teacher model parameter updating:
[0175] Exponential moving average update on the teacher model parameters:
[0176]
[0177] Wherein, a is the EMA decay factor, usually set to 0.999; denotes the model parameters obtained by the t-th update of the teacher model, denotes the model parameters obtained by the t-th update of the student model.
[0178] The above EMA mechanism can ensure the stability of the teacher model parameters, avoid parameter oscillation in the training process, and improve the model convergence.
[0179] It should be noted that steps S100-S600 can be periodically repeated according to actual application requirements to update the federated model.
[0180] It can be understood that the federated multi-modal learning method provided by the embodiment of the application proposes an alignment enhancement fusion framework for federated multi-modal learning with limited labeled data, including contrast calibration, weight enhancement and semi-supervised algorithm. The embodiment of the application realizes the extraction of unbiased representation from limited labels in a distributed and less labeled data scene, and simultaneously utilizes cross-modal complementary information, which can effectively alleviate feature bias, significantly optimize feature space, enhance multi-modal fusion, and achieve performance comparable to centralized training.
[0181] In order to verify the effectiveness of the federated multi-modal learning method provided by the embodiment of the application, refer to Figure 4The performance of the method and the existing baseline method is systematically compared on three mainstream self-supervised learning frameworks (CMC, Cosmo, FOCAL) and three standard multi-modal datasets. To ensure a fair comparison, all algorithms are trained with a unified learning rate to reach convergence. The results show that the method is significantly better than the federated learning baseline in most cases, and can even match or even surpass the performance of centralized training. It is worth noting that under the CMC self-supervised framework, the method shows a clear advantage over centralized training. This may be due to the lack of an explicit modality alignment module in CMC, which enables the alignment-enhanced fusion mechanism of the method to more effectively utilize the cross-modal synergistic effect. In addition, on the PAMAP2 dataset, the accuracy of the method is improved by 22.08%, and the F1 score is improved by 23.50%, highlighting its advantage over traditional federated learning in handling complex multi-modal data. Overall, the average accuracy of the method is improved by 15.16%, and the F1 score is improved by 17.78%, confirming its technical advantage in the task of label-constrained federated multi-modal learning.
[0182] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present specification, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0183] Although the embodiments of the present disclosure have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and purposes of the present disclosure, and the scope of the present disclosure is defined by the claims and their equivalents.
Claims
1. A federated multimodal learning method based on alignment-enhanced fusion, characterized in that: include: Step S100: Using the feature encoders of each client participating in federated learning, perform feature extraction and dimension alignment on the local multimodal data to obtain projection features; The local multimodal data is divided into local labeled multimodal data and local unlabeled multimodal data; Step S200: generating fusion features based on the projection features of the local labeled multimodal data and the local unlabeled multimodal data of each client and constructing a pseudo label set; Step S300: Perform local training in two stages on each client; In the first stage, the feature encoder is pre-trained using self-supervised contrastive learning using local unlabeled multimodal data; In the second stage, supervised contrastive learning is performed on the pre-trained feature encoder using local labeled multimodal data and pseudo-label sets to obtain the trained local feature encoder. Step S400: Aggregate the local feature encoder parameters of each client in the cloud and update the global feature encoder, and send the updated global feature encoder parameters to each client.
2. The federated multimodal learning method according to claim 1, wherein There are K clients participating in federated learning. For client k, step S100 includes: Step S110: Client k receives local n k multimodal data samples, let the received i local multimodal data samples be Among them, J stands for The number of different modal data types contained in, j represents The modal number of the data in the Step S120: The feature encoder of client k generates the feature of each modal data. Perform feature extraction: in, Represents the local multimodal data samples extracted by the feature encoder of client k The characteristics of the j-th modal data in ; Flatten represents the flattening operation of the multidimensional matrix; f encj Represents the feature encoder for the j-th modality data; Step S130: Combine features of different dimensions Map to a unified dimension d to obtain projection features Projection Features According to the following formula: in, Represents the characteristics of different dimensions Transformation operation mapped to uniform dimension d; Norm represents normalization operation.
3. The federated multimodal learning method according to claim 1, wherein: Step S200 includes: Step S210: Client k processes each local multimodal data sample The projection features are randomly fused and enhanced P times: in, For local multimodal data samples The features after the p-th fusion enhancement; α jp is a random weight coefficient, satisfying ∑α jp =1; Step S220: Construct a feature matrix with labeled data according to the following formula: and the feature matrix of unlabeled data in, express The Ljth modal data with local labels The corresponding n-th labeled fusion enhanced feature, express The Uith modality data in the local unlabeled The corresponding m-th unlabeled fusion enhanced feature; Step S230: Calculate the cosine similarity between the unlabeled features and the labeled features For each unlabeled feature The labeled feature with the highest cosine similarity is selected as the candidate labeled feature, and the label corresponding to each candidate labeled feature whose cosine similarity exceeds the preset threshold is assigned to the local unlabeled multimodal data to form a pseudo label set D * .
4. The federated multimodal learning method according to claim 1, wherein: Step S300 includes: Step S310: In the first stage, the client pre-trains the local feature encoder using local unlabeled multimodal data and self-supervised contrastive learning; Step S320, the second stage, using local labeled multimodal data and pseudo label set D * Perform supervised contrastive learning on the pre-trained local feature encoder and calculate the contrastive fusion loss L mix : Where i' is D * The sample number in ; D * The fusion feature of the sample numbered i' in the The index set of samples with the same label; Represents the fusion features of the sample with sequence number p' in P(i'); represents the fusion feature of the sample with sequence number s in P(i'); D * / P(i') is the * The set obtained by removing P(i'); a is D * / The sample number in P(i'); Indicates D * The sequence number in / P(i') is the fusion feature of sample a; τ represents the temperature coefficient.
5. The federated multimodal learning method according to claim 1, wherein: Step S400 includes: The cloud uses the trained local feature encoder parameters θ uploaded by each client. k Perform federated average aggregation to obtain the updated global feature encoder parameters θ global , the updated global feature encoder parameters are sent to each client; wherein, the updated global feature encoder parameters θ global According to the following formula: Among them, θ global is the updated global parameter; K is the number of clients participating in federated learning; n k is the number of local multimodal data samples contained in client k, and N is the total number of samples participating in federated training.
6. The federated multimodal learning method according to any one of claims 1 to 5, characterized in that: After repeatedly executing steps S100 to S400 for several times until the set upper limit of the number of iterations is reached, the federated multimodal learning method further includes: step S500, performing weighted enhanced contrast fusion in the cloud, including: Step S510: The cloud receives the deep multimodal features (z1, ..., z2) extracted by the trained local feature encoder uploaded by each client. j ,…,z J ), thereby constructing a feature set D containing labels L and the unlabeled feature set D U The global feature set of Step S520: for each deep multimodal feature z j , by fusing dynamic weight w(e) j and mass weight w(hp) j Get z j The weight W j , the dynamic weight w(e) j Calculated based on the attention mechanism, the quality weight w(hp) j Used to evaluate the clustering quality of unlabeled data; Step S530: for the i-th local multimodal data sample Utilize the deep multimodal features of the sample and its corresponding weight W j Get the multimodal fusion feature v of the sample i ; Step S540: Use the labeled features in the cloud to perform supervised contrastive learning. The loss function used is L surp : Among them, D L is the set of all labeled fusion features; i” is D L The sample number in ; D L The fusion features of the sample numbered i” in A collection of feature indexes with the same label; is the fusion feature of the sample with sequence number s in S(i”); D L / S(i”) is from D L The set obtained by removing S(i”); a’ is D L / Sample number in S(i”); D L / S(i”) is the fusion feature of sample a’; τ is the temperature coefficient; Step S550: Update weight parameters by gradient backpropagation; Step S560: Calculate the multimodal fusion feature using the updated weights as the optimized multimodal fusion feature.
7. The federated multimodal learning method according to claim 6, wherein: Weight W j The calculation formula is as follows: W j =(1-λ)w(e) j +λw(hp) j μ j =tanh(W·z j +b) Among them, μ j is the intermediate variable; tanh() is the activation function; (W is the neural network weight; b is the bias; λ is the hyperparameter; hpx is the Hopkins statistic coefficient of the jth mode.
8. The federated multimodal learning method according to claim 6, wherein: The federated multimodal learning method further includes step S600, performing multimodal semi-supervised learning in the cloud, including: Step S610: generating strong and weak data pairs required for semi-supervised learning based on the weight information obtained in step S500, for simulating the original-transformed data pair relationship in data enhancement; Step S620: deploying the teacher model and the student model on the cloud and initializing them, and using the strong and weak data pairs to simultaneously train the teacher model and the student model to implement semi-supervised learning; Step S630: updating the student model parameters based on consistency regularization and updating the teacher model parameters based on the exponential mean index to maintain the consistency between the teacher model and the student model; Step S640: The cloud sends the final teacher model parameters as global feature encoder parameters to each client.
9. The federated multimodal learning method according to claim 8, wherein: In step S610, based on the weight W obtained in step S500 j Generate strong data v according to the following formula strong and weak data v weak : Among them, Concat represents the connection operation; represents the jth deep multimodal feature of the i-th local multimodal data sample; J is the number of types of different modal data contained in the i-th local multimodal data sample.
10. The federated multimodal learning method according to claim 8, wherein: Step S620 includes: Step S621, initialize the student model S_model and the teacher model T_model; Step S622, forward propagation: Strong data v strong Input the student model and get the strong data v strong The predicted label y s ,y s =S_model(v strong ); The weak data v weak Input the teacher model and get the weak data v weak The predicted label y t ,y t =T_model(v weak ); Step S623: Total loss L total calculate: L total =L sup +λ cons *L cons L sup =CrossEntropy(y s ,y true ) L cons =MSE(y s ,y t ) Among them, L cons is a regularization function of MSE; L sup is the loss value obtained by supervised learning; CrossEntropy is the cross loss entropy function; y true is the true sample label; cons is a hyperparameter; In step S630, the total loss L is calculated based on consistency regularization. total Gradient, use backpropagation to update the student model parameters; update the teacher model parameters by exponential moving average: Among them, α is the attenuation factor of the exponential mean indicator; Represents the model parameters obtained by the teacher model t-th update, Represents the model parameters obtained by the t-th update of the student model.