A training method for a multimodal federated learning model with missing client modalities

Abstract

This invention relates to the field of machine learning technology, and more particularly to a method for training a multimodal federated learning model for client-side missing modalities. The method involves a server initializing a global model containing learnable compensation vectors. During local training on the client, modal states are identified based on binary masks. For missing modalities, the compensation vectors are introduced as surrogate features, and semantic information is extracted from other available modalities using a self-masking interaction mechanism to generate reconstructed features. The feature fusion ratio is dynamically adjusted through adaptive gated residual fusion. This invention introduces dual alignment constraints—instance-level contrastive loss and distribution-level maximum mean difference loss—to suppress client drift caused by modality missingness and data heterogeneity from both micro-semantic and macro-statistical perspectives. This invention eliminates the need for complex data generation and reconstruction, significantly reducing the computational and communication overhead of edge devices and improving the prediction accuracy, robustness, and training stability of the global model in heterogeneous environments with missing modalities.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method for training a multimodal federated learning model for client-side missing modalities. Background Technology

[0002] With the rapid development of IoT technology, smart terminals, and wearable devices, multimodal data such as images, voice, text, and physiological signals are widely collected and used in applications such as medical and health monitoring, autonomous driving, affective computing, and emergency event analysis. Multimodal learning technology, by jointly modeling information from different modalities, can fully explore the complementary relationships between modalities, thereby improving the model's feature representation ability and prediction accuracy in complex tasks, and has become one of the important research directions in the field of artificial intelligence. However, in scenarios with high requirements for privacy protection and compliance, such as medical, financial, and personal terminal devices, directly collecting multimodal data centrally for unified training is often difficult to implement. Federated learning, as a distributed machine learning paradigm, achieves multi-party collaborative modeling by exchanging only model parameters or intermediate information without sharing the original data, effectively alleviating data privacy and compliance issues. Based on this, multimodal learning is combined with federated learning to form multimodal federated learning technology, in which each client independently trains a model based on local multimodal data, and the server aggregates the client model parameters to obtain a global model.

[0003] Existing multimodal federated learning methods typically rely on the assumption that each client possesses a complete and consistent set of modal data. Under this assumption, clients locally concatenate, weight, or fuse features from different modalities, while the server aggregates the model parameters uploaded by the clients to achieve cross-client multimodal collaborative modeling. However, in real-world applications, due to differences in terminal device hardware configurations, inconsistent sensor deployments, different acquisition strategies, or device malfunctions, different clients often possess only partial modal data, resulting in widespread modality gaps at the client level. For example, in healthcare monitoring scenarios, some terminals may only collect physiological signals while lacking behavioral or image modalities; in intelligent sensing or IoT scenarios, the types of sensors configured in different devices also vary significantly. These circumstances make existing multimodal federated learning methods based on the "complete modality" assumption difficult to apply directly, severely impacting model performance and training stability.

[0004] To address the client-side missing modality problem, existing technologies mainly include two types of solutions: padding and non-padding.

[0005] Imputation schemes fall into two categories: The first is a simple imputation strategy, such as zero-value imputation or mean imputation. This involves replacing the feature vectors of the missing modality with fixed values ​​locally on the client side to maintain the consistency of the model's input dimensions. While simple to implement, this approach fails to consider the semantic correlations between different modalities, easily introducing irrelevant or distorted feature information into the model, thus affecting feature representation quality and model training performance. The second is a generative reconstruction imputation strategy. This approach reconstructs the features or data of the missing modality by building a generative model, such as conditional generative networks, variational autoencoders, or other cross-modal modeling methods, attempting to infer the content of the missing modality using information from existing modalities. Although this approach can restore intermodal correlations to some extent, it typically relies on generative models with large parameter sizes, resulting in high computational complexity and communication overhead. Furthermore, it is difficult to deploy efficiently on resource-constrained edge devices and in federated learning scenarios, and may also face problems such as training instability and generated features deviating from the true distribution.

[0006] The second approach is the non-filling scheme, which attempts to bypass explicit reconstruction by using graph structures, prototype alignment methods, or other methods to compensate for missing features by establishing associations in the representation space. Graph structure methods capture relevance by constructing cross-modal or cross-client graphs, but these methods typically rely on complete cross-modal relationships or global graph structures, potentially impacting privacy boundaries in federated environments. They require knowledge of each client's modal composition and struggle to adapt to dynamically changing client modal states. Another mainstream approach is alignment strategies based on prototypes or knowledge distillation, such as using server-side class prototypes to assist client training, fusing multimodal features through knowledge distillation, or employing lightweight early fusion based on shared projection matrices. However, these methods primarily focus on label-level global relationship alignment or feature-level knowledge transfer, lacking structural modeling and explicit compensation mechanisms for modality missing features themselves. When modality is severely missing, feature or prototype alignment alone is insufficient to fill the fundamental information gap. While some advanced feature fusion methods introduce dynamic fusion strategies such as attention mechanisms, they often still employ static or fixed fusion strategies when dealing with missing modalities, lacking adaptive adjustment mechanisms for differences in modality integrity. Meanwhile, it ignores the drift problem of different modalities at the statistical distribution level, resulting in limited fusion performance and model stability in complex federated environments with non-independent and identically distributed modalities and missing modalities. Furthermore, while some existing methods introduce feature alignment or distribution alignment mechanisms, most only constrain multimodal features at a single level, making it difficult to simultaneously consider semantic consistency at the instance level and statistical consistency at the overall distribution level. This fails to effectively alleviate the client-side drift problem caused by missing modalities and client heterogeneity.

[0007] In summary, existing multimodal federated learning techniques generally suffer from the following shortcomings when addressing client-side modality missingness: simple imputation methods ignore semantic relationships between modalities and are prone to introducing noise; generative reconstruction methods have high computational and communication costs, limiting their practicality; and non-imputation methods lack direct compensation mechanisms for modality missingness, making it difficult to maintain model stability and robustness in complex heterogeneous environments. Therefore, there is an urgent need for a multimodal federated learning method that can effectively compensate for client-side missing modalities in a lightweight manner while ensuring privacy and security, and also taking into account cross-modal interaction modeling and feature alignment. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a method for training a multimodal federated learning model for client-side missing modalities, comprising the following steps:

[0009] S1. Initialize the system model and construct a global multimodal model for client-side missing modalities. Based on the server side, initialize the global model parameters and set the number of multimodal federated learning communication rounds, client sampling ratio, and loss function weight parameters according to the requirements of the target learning task.

[0010] S2. Based on the client sampling ratio, extract the clients participating in the current training round from all clients, construct the training subset, and broadcast the current global model parameters to each client in the training subset;

[0011] S3. Based on the global model parameters, perform local model initialization for each client in the training subset, and combine the local multimodal dataset and loss function weight parameters of each client to perform independent local training for each client, and obtain the local parameters updated after independent local training for each client.

[0012] S4. Upload the updated local parameters to the server and aggregate the updated local parameters using a federated average algorithm based on the amount of client data to construct a new round of global model parameters;

[0013] S5. Based on the new round of global model parameters, repeat steps S2-S4 until the preset number of multimodal federated learning communication rounds is reached to obtain the trained global multimodal model.

[0014] A computer-readable storage medium for storing a computer program that, when run on a computer, causes the computer to perform any of the above-mentioned methods for training a client-side missing modality multimodal federated learning model.

[0015] An electronic device includes: a memory for storing a computer program; and a processor for executing the computer program to implement a client-side missing modality multimodal federated learning model training method as described above.

[0016] The beneficial effect of this invention is that it provides a lightweight, non-generative missing modality compensation scheme. When modality is missing on the client side, it utilizes the semantic information of available modalities to introduce learnable compensation vectors as surrogate features for the missing modalities, thereby filling the modality information gap without complex data reconstruction. This approach avoids the introduction of large-scale generative models, reduces the computational complexity of local training on the client side, and the overhead of cross-client communication. This makes the method more suitable for deployment in edge devices or IoT terminals with limited computing, storage, and communication resources, and helps improve the feature representation quality and overall model performance in modality-missing scenarios.

[0017] Simultaneously, by constructing a dynamic fusion mechanism based on semantic relevance, the similarity relationship between different modal features is calculated in the projection space, and a self-masking operation is introduced to form a semantic relationship matrix between modalities. This ensures that the model only obtains complementary information from other modalities during the fusion process, avoiding over-reliance on its own modal features. Building upon this, a lightweight adaptive gating unit dynamically adjusts the proportion of supplementary information introduced from other modalities based on the information reliability of each modal feature, achieving sample-level adaptive feature enhancement and cleansing. Furthermore, by introducing a combination of instance-level feature alignment constraints and distribution-level statistical alignment constraints, the semantic consistency of different modal features within the same sample is constrained at the micro level, and the differences between the overall distributions of different modal features are reduced at the macro level. This mitigates the feature drift problem caused by data heterogeneity and modality missing features, providing support for stable aggregation in the federated learning process.

[0018] In summary, through the synergistic effect of the above technical solutions, the server can aggregate a global model suitable for multimodal data and capable of handling different combinations of missing modalities. This model, without assuming complete client modalities, can effectively model input data with arbitrary modality missingities, maintaining good predictive stability and robustness. Therefore, it provides a privacy-preserving, secure, and easily deployable multimodal federated learning technical solution for practical applications such as medical diagnosis, intelligent sensing, and human-computer interaction. Attached Figure Description

[0019] Figure 1 This is a flowchart of a multimodal federated learning model training method for client-side missing modalities proposed in an embodiment of the present invention.

[0020] Figure 2This is a schematic diagram of the terminal device structure of a multimodal federated learning model training method for client-side missing modalities proposed in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of a computer-readable storage medium structure for a multimodal federated learning model training method for client-side missing modalities proposed in an embodiment of the present invention.

[0022] In the diagram, 200 is the terminal device, 210 is the memory, 211 is the RAM, 212 is the cache memory, 213 is the ROM, 214 is the program / utility, 215 is the program module, 220 is the processor, 230 is the bus, 240 is the external device, 250 is the I / O interface, 260 is the network adapter, and 300 is the program product. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, and not all of them. The components of the embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0025] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or machine that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or machine. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or machine that includes said element.

[0026] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0027] Example 1:

[0028] like Figure 1 As shown, this embodiment provides a method for training a multimodal federated learning model for client-side missing modalities, including the following steps:

[0029] S1. Initialize the system model and construct a global multimodal model for client-side missing modalities. Based on the server side, initialize the global model parameters and set the number of multimodal federated learning communication rounds, client sampling ratio, and loss function weight parameters according to the requirements of the target learning task.

[0030] S2. Based on the client sampling ratio, extract the clients participating in the current training round from all clients, construct the training subset, and broadcast the current global model parameters to each client in the training subset;

[0031] S3. Based on the global model parameters, perform local model initialization for each client in the training subset, and combine the local multimodal dataset and loss function weight parameters of each client to perform independent local training for each client, and obtain the local parameters updated after independent local training for each client.

[0032] S4. Upload the updated local parameters to the server and aggregate the updated local parameters using a federated average algorithm based on the amount of client data to construct a new round of global model parameters;

[0033] S5. Based on the new round of global model parameters, repeat steps S2-S4 until the preset number of multimodal federated learning communication rounds is reached to obtain the trained global multimodal model.

[0034] Furthermore, the global model parameters mentioned in step S1 include model structure parameters and learnable compensation vector parameters. The model structure parameters include feature extractor parameters, feature fusion module parameters, and decoder parameters. The learnable compensation vector parameters are embedded in the global model parameter set in the form of trainable model parameters. Their dimension is consistent with the high-dimensional semantic feature representation of the corresponding modality. When a certain modality data is missing in the terminal device, the system uses the corresponding learnable compensation vector in the feature space to replace the feature vector of that modality, so that it can participate in subsequent cross-modal semantic fusion, relation modeling, and classification decision calculation. At the same time, the compensation vector, as a learnable parameter of the model, is updated by gradient on the client side according to the task loss and regularization loss. On the server side, the updated parameters uploaded by each client are weighted and fused through a federated aggregation mechanism, so as to gradually converge to the optimal feature proxy representation of the missing modality that adapts to the global statistical distribution during the multi-client collaborative optimization process.

[0035] Specifically, in step S1, the server first randomly initializes the global model parameters. The global model parameters include model structure parameters shared by the server and all clients, as well as learnable compensation vector parameters set according to modality. The model structure parameters include at least: feature extractor parameters for extracting features from different modalities, feature fusion module parameters for achieving cross-modal semantic interaction and fusion, and decoder parameters for completing downstream tasks. The server-side simultaneously sets the number of communication rounds for federated training, the client sampling rate, and the weight parameters for various loss functions, and uses these parameters as the initial system configuration, setting the total number of training communication rounds to [number missing]. .

[0036] Furthermore, step S2 includes the server randomly selecting all registered clients based on a preset client sampling ratio to form the training subset for the current round, and then distributing the current global model parameters to each client in the training subset.

[0037] Specifically, for each communication round The server randomly selects a subset of all registered clients based on a preset sampling rate to form the training subset for the current round. Subsequently, the server will set the current global model parameters. Distribute to subsets Each client in the process.

[0038] Furthermore, step S3 includes the following sub-steps:

[0039] S301. Each client in the training subset receives global model parameters to initialize its local model, and constructs a set of available modalities and a set of missing modalities based on the local multimodal dataset of each client.

[0040] S302. Using a feature extractor, each modality in the available modality set is mapped to a high-dimensional semantic feature space to obtain the true feature representation; based on the learnable compensation vector parameters, a learnable compensation vector corresponding to the missing modality set is constructed as a surrogate feature representation, and combined with the true feature representation to construct a unified feature set;

[0041] S303. Semantic relation modeling is performed based on the unified feature set to obtain a semantic relation matrix. Based on the semantic relation matrix, a cross-modal reconstruction feature set corresponding to the unified feature set is generated through a self-masking interactive weight algorithm. The unified feature set and the cross-modal reconstruction feature set are residually fused through an adaptive gating unit to obtain a fused feature set.

[0042] S304. Based on the fused feature set, instance-level semantic alignment and distribution-level statistical alignment are performed respectively using the double alignment regularization algorithm to obtain the difference between the instance-level contrast loss value and the distribution-level maximum mean value. Combined with the weight parameters of the loss function, the local total loss value is calculated.

[0043] S305. Backpropagate the local total loss value and update the local model parameters of each client using the stochastic gradient descent optimizer to obtain the local parameters updated after independent local training for each client.

[0044] Specifically, the implementation principles of the above embodiments and their respective sub-steps are as follows:

[0045] Each selected client After receiving the global model parameters from the server, the local training process is executed independently, which includes:

[0046] First, perform local model initialization and data preparation on the client side. Receive global model parameters sent by the server Initialize the local model and load its local multimodal dataset. Specifically, Represents the complete set of modalities in the dataset. It is a set of available modes. It is a client with a missing modality set. Based on its own available modality set, it constructs corresponding modality availability identification information for each sample. The available modalities can be represented as follows: , This indicates the missing modality. A corresponding binary mask is also prepared for each sample. .

[0047] Then, modal feature extraction and missing modality compensation are performed. For each training sample on the client's local machine:

[0048] For the sample Each mode that actually exists in its original data Input to the corresponding modality-specific feature extractor The extractor maps the raw data to a high-dimensional semantic feature space, obtaining the true feature representation: ,in It is a preset feature dimension. When data of a certain modality is missing in the sample, no data reconstruction or generation operation is performed. Instead, the learnable compensation vector corresponding to that modality is directly introduced as the surrogate feature of that modality.

[0049] For the sample Each missing mode This embodiment does not perform complex data generation or reconstruction, but instead introduces a globally learnable compensation vector for different modalities. This vector, as part of the model parameters, is continuously optimized via gradient descent during federated training. Its goal is to learn to become a "useful" semantic placeholder or surrogate feature in subsequent cross-modal interaction and alignment tasks. Employing lightweight parameterized vector compensation instead of complex generative reconstruction significantly reduces the computational complexity and communication burden of local training on the client side, enabling the method to run efficiently on resource-constrained edge devices.

[0050] Next, cross-modal semantic interaction and adaptive fusion are performed. Based on the unified feature set, the client performs cross-modal semantic interaction and fusion operations on each modal feature.

[0051] Then, loss calculation is performed, which includes calculating task loss and calculating local total loss. The task loss is calculated as follows. At that time, the double alignment regularization module is invoked to calculate the instance-level contrastive loss. and the maximum mean difference of the distribution level The task loss is obtained; the total local loss can be expressed as: ,in and These are the preset hyperparameters.

[0052] Then, backpropagation and parameter updates are performed to obtain the local total loss. Perform backpropagation and update all parameters of the local model using optimizers such as stochastic gradient descent. Obtain the updated local parameters. .

[0053] Finally, the client Updated local parameters Uploaded to the server.

[0054] Furthermore, step S303 also includes the following sub-steps:

[0055] S303-1. Map each modal feature to a common projection space using a projection head, calculate the cosine similarity of all modal pairs in the current sample, and construct a semantic relationship matrix;

[0056] S303-2. Perform temperature scaling and self-masking operations on the semantic relation matrix, set the diagonal elements to negative infinity and normalize them to generate the interaction weight matrix, and calculate the cross-modal reconstruction features for each modality.

[0057] S303-3. The gating unit dynamically adjusts the gating coefficient based on the reliability of the input feature information, and uses the residual form to fuse the original features with the cross-modal reconstructed features to obtain a fused feature set.

[0058] Specifically, the implementation principles of the above embodiments and their respective sub-steps are as follows:

[0059] First, through a set of lightweight projection heads ,in Indicates the number of modes, and the features of each mode. Mapping to a common, low-dimensional projection space to enable cross-modal comparisons and obtain the projected features: ;in, Indicates the dimension of the projection space.

[0060] Next, the cosine similarity of all modality pairs in the current sample is calculated, constructing a dimension of... semantic relation matrix Specifically, it is expressed as:

[0061] ;

[0062] in, The cosine similarity function is used. This represents the matrix transpose operation, where each element of the matrix... Explicitly quantified the modes With mode The strength of association in the current semantic context.

[0063] Then, in order to force the model to interact across modalities and avoid a particular modality from becoming overly dependent on itself (especially when it is a feature generated through compensation), this embodiment is based on a relation matrix. Calculate an interaction weight matrix Specifically, let's first... Temperature scaling is performed and self-masking is applied, setting the diagonal elements (i.e., the similarity between each mode and itself) to negative infinity. , to obtain the matrix Each row can be represented as: ;in Temperature hyperparameters are used to control the smoothness of the distribution and adjust the sharpness of the distribution.

[0064] Then, to each line Normalization is performed using the Softmax function. Since the diagonal is... After softmax, the corresponding weights are 0. The final interaction weight matrix is ​​obtained. Its elements Indicates to enhance or supplement the modality It should be from the modal How much information to absorb, and what are the requirements? . use Calculate the cross-modal reconstruction features for each mode. It is a weighted sum of all other modal features: This feature can be viewed as a mode-specific feature provided collaboratively by other modes. Context-aware supplementary information.

[0065] In obtaining the original features and reconstruct supplementary features Subsequently, this embodiment designs an adaptive gated residual fusion mechanism to dynamically determine the fusion ratio. A lightweight gating unit (e.g., a small two-layer one-dimensional convolutional network) uses the concatenated features... As input, the sigmoid activation function outputs a scalar gating coefficient. .

[0066] Ultimately, modality Features after fusion Calculated using residual form: ;in This indicates element-wise multiplication. This can be viewed as "information residuals" transmitted from other modalities. Gating coefficient. Based on the modes in the current sample Adaptive adjustment of information reliability: if If it comes from real data and is rich in information, then The model tends to retain the original features; if If the compensation vector or information is insufficient, then The model incorporates more supplementary information from other modalities. Ultimately, the cross-modal semantic interaction fusion module outputs a unified fusion feature set after semantic enhancement and adaptive cleansing. ;in, This represents the final feature of the first mode after adaptive residual fusion. This represents the final feature of the second mode after adaptive residual fusion. Indicates the first The final features of each modality after adaptive residual fusion. The self-masking mechanism forces semantic complementarity across modalities, effectively utilizing information-rich modalities to enhance "weak" features generated by compensation vectors. Combined with adaptive gating, sample-level dynamic fusion is achieved, significantly improving the accuracy of feature fusion and the robustness of the model in the case of modality loss.

[0067] Furthermore, the instance-level semantic alignment in step S304 includes narrowing the distance between different modal features of the same sample in the semantic space and widening the feature distance between different samples by using contrastive learning loss; the distribution-level statistical alignment includes minimizing the difference in the overall statistical distribution of different modal features within a batch by using maximum mean difference loss.

[0068] Specifically, the purpose of instance-level semantic alignment is to shorten the distance between different modal features of the same sample in the semantic space, and to widen the distance between features of different samples. Specifically, in a training batch, the same data samples... Different modal features are considered as positive sample pairs, and all features of this sample and other samples in the batch are considered as negative sample pairs. This embodiment applies this contrastive loss at two levels: calculating the loss on the output true features and the compensation vector. The loss is calculated on the projection of the true output features and the compensation vector. The instance-level contrastive loss is calculated, and its formula is expressed as:

[0069] ;

[0070] ;

[0071] ;

[0072] in, For batch size, and Each represents the same sample In two different modes The following features constitute positive sample pairs. Represents any sample in the current batch arbitrary modes The characteristics, when hour, To form negative sample pairs, and Each represents the same sample In two different modes Projection of features below, Represents any sample in the current batch arbitrary modes The projection of the features, The cosine similarity function is used. It is an exponential function. Temperature hyperparameters are used to control the smoothness of the distribution.

[0073] Specifically, the goal of distribution-level statistical alignment is to minimize the differences in the overall distribution of different modal features, not just the differences in individual sample pairs. Specifically, at the batch level, for the client... Any two different modes exist in and Calculate their features after feature extraction or compensation. and The maximum difference in mean between them is calculated using the following formula:

[0074] ;

[0075] in, It maps features to the reproducing kernel Hilbert space ( The kernel function of ).

[0076] Instance-level alignment ensures fine-grained semantic consistency, while distribution-level alignment ensures macroscopic statistical consistency. The combination of these two approaches creates a stable and alignable local model update mechanism for subsequent federated aggregation.

[0077] Furthermore, in step S4, the server receives all participating clients. After uploading local parameters, a client-based approach is adopted. Data size The aggregation strategy uses a weighted average, and the federated average algorithm based on the amount of client data is expressed as follows:

[0078] ;

[0079] Where t represents the current training round, and t+1 represents the new training round. This represents the global model parameters for the new training epoch, where k represents the client. This indicates the subset of participants in the training, where j represents the j-th client in the subset. Indicates the amount of data on the client side. This represents the updated local parameters uploaded by the k-th client.

[0080] This yields a new set of global model parameters. Finally, repeat steps 2 through 4 until the preset number of communication rounds is reached. Finally, the server receives the trained global model. This model can be deployed for inference and prediction on new data that may have arbitrary combinations of missing modalities. By imposing constraints at both the micro and macro levels through dual alignment, "client drift" caused by missing modalities and data heterogeneity is greatly suppressed, improving the convergence speed and stability of federated training. Finally, based on the method described in this embodiment, a global framework with stronger generalization ability is obtained.

[0081] Example 2

[0082] Based on Example 1, there is a training scenario for a multimodal federated learning model with missing client modalities. This scenario adopts a training method for a multimodal federated learning model with missing client modalities as described in the above examples. Specifically, the missing client modalities in this scenario are physiological signal recognition for multiple users. It is assumed that there are three clients (users) participating in the training, and their available sensor modalities (i.e., data modalities) are electrocardiogram (ECG) signals and electromyogram (EMG) signals. The task objective is to classify the user's movement state (such as resting, walking, and running) based on these two signals.

[0083] Specifically, it includes the following steps:

[0084] Step 1: System Initialization and Data Preparation

[0085] 1. Server-side initialization:

[0086] The server randomly initializes a global multimodal model, whose parameters are denoted as follows: The model contains the following core components:

[0087] Modality-specific feature extractor: two independent feature encoding networks, Used for processing electrocardiogram signals. Used for processing electromyographic signals, the output feature dimension is 1. .

[0088] Learnable compensation vector: A learnable parameter vector is pre-defined for each mode, serving as a surrogate feature when that mode is missing. Let the ECG signal compensation vector be... The electromyographic signal compensation vector is .

[0089] Cross-modal semantic interaction and fusion module: includes a projection head (mapping 128-dimensional features to a 64-dimensional common space), a self-masking relation matrix calculation unit, and an adaptive gated residual fusion unit.

[0090] Task decoder: A fully connected classifier that maps the fused 256-dimensional features to three motion state categories.

[0091] Server settings for federated learning hyperparameters: Total number of communication rounds Client sampling ratio in each round Instance-level comparative loss weights Distributed MMD loss weights Temperature overparameter =0.1.

[0092] 2. Client data and missing settings:

[0093] Assume the local data modalities of three clients (A, B, C) are as follows, to simulate real modality loss:

[0094] Client A: Only ECG signal data was collected; electromyography (EMG) signals are missing. Its local dataset is... ;in, This represents the local electrocardiogram (ECG) signal data collected by client A. Indicates sample Corresponding category tags;

[0095] Client B: Only electromyography (EMG) signal data was collected; electrocardiogram (ECG) signals are missing. Its local dataset is... ;in, This represents the local electromyography (EMG) signal data collected by client B. Indicates sample Corresponding category tags;

[0096] Client C: Simultaneously acquired both ECG and EMG signals, with complete modalities; its local dataset is as follows. ,in, This represents the local electrocardiogram (ECG) signal data collected by client C. This represents the local electromyography (EMG) signal data collected by client C. Indicates sample The corresponding category label.

[0097] Each client generates a modal availability mask for each of its samples. For example, the sample mask for client A is... (ECG present, EMG absent).

[0098] Step Two: Example of a Single Round of Federated Training (using the example of the first round of training) (Taking a round as an example) Suppose that in this round, the server randomly selects client A and client C to participate in training.

[0099] 1. Global Model Broadcast: The server broadcasts the current global model parameters. Send to clients A and C;

[0100] 2. Local training on the client (taking client A as an example):

[0101] First, local initialization and data preparation are performed: Client A receives... Initialize the local model. Load its local dataset. And prepare a mask for each sample. ;

[0102] Then, modal feature extraction and missing data compensation are performed: for existing ECG modalities, the original data is input. To feature extractor To obtain the true features For missing electromyographic modalities, no data reconstruction is performed; instead, the learnable compensation vectors in the global model are used directly. As the electromyographic proxy feature of this sample, i.e. ;

[0103] Next, cross-modal semantic interaction and adaptive fusion, specifically including:

[0104] Semantic relation modeling: and Projected onto a 64-dimensional public space respectively, we obtain and Calculate the cosine similarity between the two and construct a 2x2 relationship matrix. ;

[0105] Self-masking interaction weight calculation: For By applying a self-masking operation, setting the diagonal elements (their own similarity) to negative infinity, and then performing Softmax normalization on each row, the interaction weight matrix is ​​obtained. This results in the weights of ECG features being fully allocated to EMG (compensated) features during fusion, and vice versa;

[0106] Adaptive residual fusion: Calculates "cross-modal reconstruction features" for each modality, such as the reconstruction features of an electrocardiogram. , is represented as: ,in, This represents the semantic contribution weight of the electromyographic modality (emg) to the reconstruction of the electrocardiographic modality (ecg). Subsequently, a lightweight gating unit is used to generate gating coefficients based on the input features. The final fusion feature calculation is as follows: Since client A's ECG data is a real feature and the information is reliable, the gating coefficient... The information may be relatively small, and the fused features mainly retain their own information; however, their electromyography (EMG) is a compensation vector with weaker information and gating coefficients. It may be relatively large, thus affecting its fusion characteristics. It relies more on information reconstructed from real electrocardiogram characteristics.

[0107] Next, calculate the task loss. The fused bimodal features are concatenated and input into a classifier to obtain predictions, which are then compared with the true labels. Cross-entropy loss;

[0108] The calculation of double alignment loss specifically includes:

[0109] Instance-level alignment loss Within a batch, the true electrocardiogram characteristics of the same sample are... With electromyographic compensation characteristics projection and As a positive sample pair, it forms a negative sample pair with all feature projections of other samples in the batch. The InfoNCE contrastive loss is calculated to narrow the semantic distance between different modal features of the same sample.

[0110] Distribution-level alignment loss : Calculate the maximum mean difference loss between the true electrocardiogram feature distribution and the compensated electromyogram feature distribution of this batch, so as to macroscopically constrain the statistical properties of the compensation vector to approach the true feature distribution.

[0111] The total loss was finally calculated. , is represented as: Then, using the total loss Perform backpropagation on the client's local model parameters to update the client A's local model parameters, including the feature extractor. Fusion module parameters and the crucial electromyography compensation vector Updated local parameters Uploaded to the server.

[0112] The local training process for client C is similar, but because its modality is complete and both features are extracted from the ground truth, its training simultaneously optimizes both feature extractors and all network parameters.

[0113] 3. Server-side aggregation:

[0114] The server receives local parameters uploaded by clients A and C. and Based on the federated averaging algorithm, a weighted average is calculated according to the amount of client data to obtain the new round of global model parameters. In this round of aggregation, the electromyography compensation vector in the global model... Integrating data from client A (lacking electromyography, optimized using electrocardiogram) The knowledge of both the client C (modal complete, providing information on the true distribution of electromyographic features) and the client C (which provides information on the distribution of true electromyographic features) makes it a more effective semantic agent that can collaborate with electrocardiogram features.

[0115] Step 3: Multiple iterations and final model application:

[0116] Repeat step two for multiple rounds of federated training (e.g., 100 rounds). As training progresses, knowledge from clients with different missing modalities (e.g., A with only ECG, B with only EMG, and C with complete bimodality) is continuously integrated into the global model. Learnable compensation vectors... and Under the constraint of double alignment loss, the proxy features are optimized to effectively represent their corresponding modal semantics and can interact complementaryly with other modalities. The cross-modal interaction and adaptive fusion module learns how to dynamically adjust information flow based on modal integrity (real or compensated). Finally, the server obtains the trained global model. The model can be deployed on new users' terminal devices: whether the new device only has ECG, only has EMG, or both, the model can robustly extract and fuse features from the input signal through its embedded compensation vector and adaptive fusion mechanism, thereby achieving stable and accurate identification of the user's motion state.

[0117] Example 3

[0118] like Figure 2 As shown, this embodiment proposes a terminal device for a multimodal federated learning model training method for client-side missing modalities. The terminal device includes at least one memory, at least one processor, and a bus connecting different platform systems.

[0119] The memory may include readable media in the form of volatile memory, such as RAM 211 and / or cache memory, and may further include ROM 213.

[0120] The memory also stores a computer program that can be executed by a processor, causing the processor to perform any of the above-described methods for training a multimodal federated learning model for client-side missing modalities in this application. The specific implementation and technical effects are consistent with those described in the embodiments of the above methods, and some details will not be repeated here. The memory may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0121] Accordingly, the processor can execute the aforementioned computer programs, as well as executable programs / utilities.

[0122] A bus can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus structures.

[0123] The terminal device can also communicate with one or more external devices, such as keyboards, pointing devices, Bluetooth devices, etc., and with one or more devices capable of interacting with the terminal device, and / or with any device that enables the terminal device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed through I / O interfaces. Furthermore, the terminal device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter can communicate with other modules of the terminal device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the terminal device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0124] Example 4

[0125] like Figure 3 As shown in Example 1, this example proposes a computer-readable storage medium for a multimodal federated learning model training method for client-side missing modalities. The computer-readable storage medium stores instructions that, when executed by a processor, implement any of the aforementioned multimodal federated learning model training methods for client-side missing modalities. The specific implementation method and the achieved technical effects are consistent with those described in the examples above, and some details will not be repeated.

[0126] This embodiment provides a program product for implementing the above-described method, which may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this embodiment, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0127] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to user computing devices via any type of network, including local area networks (LANs) or wide area networks (WANs), or they can be connected to external computing devices (e.g., via the Internet through an Internet service provider).

[0128] This invention is described from the perspectives of its intended use, effectiveness, progress, and novelty. Its practical and progressive features meet the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings are merely preferred embodiments of this application and are not intended to limit this application. Therefore, all structures, devices, features, etc., that are similar to or identical to those of this application, i.e., all equivalent substitutions or modifications made in accordance with the scope of this patent application, shall fall within the scope of protection of this patent application.

[0129] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a multimodal federated learning model for client-side missing modalities, characterized in that, Includes the following steps: S1. Initialize the system model and construct a global multimodal model for client-side missing modalities. Based on the server side, initialize the global model parameters and set the number of multimodal federated learning communication rounds, client sampling ratio, and loss function weight parameters according to the requirements of the target learning task. S2. Based on the client sampling ratio, extract the clients participating in the current training round from all clients, construct the training subset, and broadcast the current global model parameters to each client in the training subset; S3. Based on the global model parameters, perform local model initialization for each client in the training subset, and combine the local multimodal dataset and loss function weight parameters of each client to perform independent local training for each client, and obtain the local parameters updated after independent local training for each client. S4. Upload the updated local parameters to the server and aggregate the updated local parameters using a federated average algorithm based on the amount of client data to construct a new round of global model parameters; S5. Based on the new round of global model parameters, repeat steps S2-S4 until the preset number of multimodal federated learning communication rounds is reached to obtain the trained global multimodal model; The global model parameters mentioned in step S1 include model structure parameters and learnable compensation vector parameters; the model structure parameters include feature extractor parameters, feature fusion module parameters, and decoder parameters. Step S3 includes the following sub-steps: S301. Each client in the training subset receives global model parameters to initialize its local model, and constructs a set of available modalities and a set of missing modalities based on the local multimodal dataset of each client. S302. Using a feature extractor, each modality in the available modality set is mapped to a high-dimensional semantic feature space to obtain the true feature representation; Based on the learnable compensation vector parameters, a learnable compensation vector corresponding to the set of missing modalities is constructed as a surrogate feature representation, and a unified feature set is constructed by combining it with the real feature representation; S303. Semantic relation modeling is performed based on the unified feature set to obtain a semantic relation matrix. Based on the semantic relation matrix, a cross-modal reconstruction feature set corresponding to the unified feature set is generated through a self-masking interactive weight algorithm. The unified feature set and the cross-modal reconstruction feature set are residually fused through an adaptive gating unit to obtain a fused feature set. S304. Based on the fused feature set, instance-level semantic alignment and distribution-level statistical alignment are performed respectively using the double alignment regularization algorithm to obtain the difference between the instance-level contrast loss value and the distribution-level maximum mean value. Combined with the weight parameters of the loss function, the local total loss value is calculated. S305. Backpropagate the local total loss value and update the local model parameters of each client using the stochastic gradient descent optimizer to obtain the local parameters updated after independent local training for each client.

2. The method for training a multimodal federated learning model for client-side missing modalities according to claim 1, characterized in that, Step S2 includes the server randomly selecting all registered clients based on a preset client sampling ratio to form the training subset for the current round, and then sending the current global model parameters to each client in the training subset.

3. The method for training a multimodal federated learning model for client-side missing modalities according to claim 1, characterized in that, Step S303 also includes the following sub-steps: S303-1. Map each modal feature to a common projection space using a projection head, calculate the cosine similarity of all modal pairs in the current sample, and construct a semantic relationship matrix; S303-2. Perform temperature scaling and self-masking operations on the semantic relation matrix, set the diagonal elements to negative infinity and normalize them to generate the interaction weight matrix, and calculate the cross-modal reconstruction features for each modality. S303-3. The gating unit dynamically adjusts the gating coefficient based on the reliability of the input feature information, and uses the residual form to fuse the original features with the cross-modal reconstructed features to obtain a fused feature set.

4. The method for training a multimodal federated learning model for client-side missing modalities according to claim 1, characterized in that, The instance-level semantic alignment described in step S304 includes using contrastive learning loss to bring the distance between different modal features of the same sample in the semantic space closer and to widen the feature distance between different samples. The distribution-level statistical alignment includes minimizing the differences in the overall statistical distribution of different modal features within a batch by using the maximum mean difference loss.

5. The method for training a multimodal federated learning model for client-side missing modalities according to claim 1, characterized in that, The federated averaging algorithm based on client data volume described in step S4 is expressed as follows: ； Where t represents the current training round, and t+1 represents the new training round. This represents the global model parameters for the new training epoch, where k represents the client. This indicates the subset of participants in the training, where j represents the j-th client in the subset. Indicates the amount of data on the client side. This represents the updated local parameters uploaded by the k-th client.

Images