A robust multimodal federated learning approach

Abstract

The present invention belongs to the field of artificial intelligence and machine learning, and discloses a robust multimodal federated learning framework, including: S1 obtains missing modalities, screens out modalities with strong correlation with the missing modalities, uses multi-layer perceptron training for prediction, and completes modal alignment; S2 labels the feature data in the modalities, and synthesizes feature vectors in combination with the feature data; S3 uses an attention-based mechanism to fuse feature data to form a modal vector; S4 performs local training on the aligned and fused modal data; S5 uploads the local training model to the server, uses a federated learning framework for aggregation operations, and distributes the model to the client for the next round of iteration until the model converges. The present invention effectively solves the problem of reduced training model accuracy when the modality is missing by predicting the missing modality and fusing it with multimodal data, reduces the impact of modality missing on training accuracy, and improves the reliability of federated learning.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and machine learning, and specifically relates to a robust multimodal federated learning method. Background Art

[0002] In recent years, the scale of the Internet of Things has been growing, and a large number of IoT devices can continuously collect multimodal data, such as text, voice, images, videos, etc. With the continuous development of artificial intelligence technology, multimodal models are considered to be one of the best ways to improve the capabilities of artificial intelligence systems. Compared with traditional single-modal models, multimodal models can use a variety of different data inputs, such as text, voice, images, videos, etc. to more comprehensively understand and process information and improve system performance. For example, by using image and voice input at the same time, the characteristics of objects and environmental backgrounds can be more accurately identified and understood, thereby improving the accuracy of image recognition and voice recognition.

[0003] Federated Learning is a distributed machine learning method that can perform collaborative training between multiple devices or nodes without centralizing data to a central server. Federated learning not only protects data privacy, but also effectively utilizes distributed resources to improve the efficiency and flexibility of model training. At present, most federated learning methods only consider scenarios where the client's private data is of the same modality. A few works attempt to use multimodal data for multimodal federated learning to improve the training accuracy of the system. However, the above work assumes that all modalities are sufficient and complete for each multimodal sample. In the real world, when the sensor that collects data encounters a series of software and hardware problems such as failure or poor contact, multimodal data will inevitably be partially lost. Due to the lack of certain modalities, the association and complementary information between different modalities cannot be fully utilized, making it difficult for the model to establish accurate feature expression and prediction capabilities, resulting in a sharp drop in training accuracy. Therefore, how to obtain a high-precision training model in the absence of multimodal data is a problem that technicians in this field urgently need to solve. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a robust multimodal federated learning method to obtain high-precision training results when modal data is missing. This method proposes an algorithm based on interpolation mode, predicts and aligns the missing modalities, uses a multi-head attention mechanism to fuse the modalities, and improves the model accuracy through federated learning, effectively reducing the impact of modal missing on system performance.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0006] The present invention is a robust multimodal federated learning method, which specifically includes the following steps: The multimodal federated learning method obtains high-precision training results in the absence of modal data, specifically including the following steps:

[0007] Step 1: For clients with missing modalities, predict the missing modal data based on the existing modal data: first, select features with a strong correlation with the missing modal data from the existing modal data, use a multi-layer perceptron to train this part of the feature data, predict the unknown modality, and complete the modal alignment operation;

[0008] Step 2: Process the features in the existing mode and the predicted missing mode: for each feature, label it according to the importance of the feature to its location, and fuse the features with the feature labels to form a feature vector;

[0009] Step 3: Fuse the modal data aligned in step 1: design a modal label matrix based on whether the modal data is predicted data and the importance of the modal data, fuse the feature vector constructed in step 2 with the modal label in the modal label matrix, obtain multi-modal weighted data, and synthesize the modal vector of each client;

[0010] Step 4: Locally train the fused multimodal weighted data: Use redundant computing resources of the client to train the multimodal weighted data. Each client locally uses a fully connected neural network model to perform training operations on the multimodal weighted data to obtain a multimodal model.

[0011] Step 5: Upload the multimodal model trained locally on each client obtained in step 4 to the server, share the training results of each client on the server, realize collective learning and optimization of the multimodal model, adopt the corresponding federated aggregation algorithm to perform aggregation operations according to actual conditions, update the global multimodal model, and redistribute the aggregated multimodal model to each client for the next round of iteration until the multimodal model converges.

[0012] A further improvement of the present invention is that in step 1, the missing modal data is predicted based on the existing modal data, which specifically includes the following steps:

[0013] Step 1.1, feature extraction: extract features that are highly correlated with the missing modal data from the existing modal data, determine whether the feature information can be used to predict the missing modality, and if the feature information is available, put it into the feature queue for predicting the missing modality;

[0014] Step 1.2, modality prediction: score and normalize the importance of the target missing modality according to the features in the feature queue to obtain the rough target modality data , after training with a multi-layer perceptron, the predicted missing mode is obtained, specifically:

[0015]

[0016] in, Represents the characteristics The rating, It means that after the filtered feature information obtains the weight, all weighted feature information is merged. Represents the normalization function, which obtains the rough target modal data by calculating the weighted feature information of the target missing modal in the existing modal , for the obtained coarse target modal data After merging, two rounds of multi-layer perceptron calculations are performed to obtain the final predicted missing mode. , the interpolation operation is completed.

[0017] A further improvement of the present invention is that the feature extraction in step 1.1 specifically includes the following steps:

[0018] Step 1.1.1: By exploring the relationship between the features of the existing modal data and the missing modal data, the features with strong correlation with the missing modal data are extracted. The feature extraction process It is expressed as:

[0019]

[0020] in, Indicates the current mode data, Indicates that from the existing mode The features extracted from

[0021] Step 1.1.2: For the extracted features, filter out the features that cannot be used for the prediction of missing modal data and discard them;

[0022] Step 1.1.3: Put the extracted features that can be used for missing modal data prediction into a feature queue for predicting missing modal data for missing modal data prediction.

[0023] A further improvement of the present invention is that step 2 specifically comprises the following steps:

[0024] Step 2.1: Extract the modal feature data of each modality from the existing modality and the predicted missing modality, such as sound, image, etc. For each modality in the existing modality and the predicted missing modality The feature samples included , according to the importance of feature samples, construct feature labels and normalize them, specifically:

[0025] in, represents the calculated feature label, Represents the specific calculation process of training the encoding network, which is used to determine the feature sample to feature label The importance of training, refer to Activation function, which maps data to the range of 0 to 1;

[0026] Step 2.2: For each mode The features of the modal feature vector are synthesized and normalized weighted feature information is concatenated to obtain a one-dimensional modal feature vector. :

[0027]

[0028] in, represents the weighting process of feature information, and the accumulation process represents the weighting of each modality. In Weighted feature information concatenation.

[0029] A further improvement of the present invention is that in step 3, the modal fusion synthesized modal vector specifically includes the following steps:

[0030] Step 3.1: Process the multimodal data. After obtaining the weights, add the bias and train it through a multilayer perceptron to obtain preprocessed multimodal data for further obtaining importance scores:

[0031]

[0032] in, The modal eigenvector After splicing, the multimodal data is the weight matrix, is the bias, and the spliced ​​multimodal data is transformed through a layer of multi-layer perceptron Converting to multimodal data ;

[0033] Step 3.2: By setting the context vector and performing normalization, the importance score of the final description modal information is obtained:

[0034]

[0035] in, is the context vector. By setting different values ​​for it, a multi-head attention mechanism is implemented to dynamically adjust the weight of the modal data. And processed multimodal data After combining the transposed matrices of Function gets normalized importance score , represents the weight of each modality, that is, the importance of the modality in the final representation, The accumulated matrix is ​​the modality label matrix;

[0036] Step 3.3: All modal eigenvectors and importance rating Combine and concatenate to obtain multimodal weighted data for training:

[0037]

[0038] in, Representing modality The importance rating of Modal The modal information of Represents all modal eigenvectors Combined with the importance score and concatenated to obtain the final multimodal weighted data .

[0039] A further improvement of the present invention is that in step 4, a fully connected neural network is established locally for the client, and the process is as follows:

[0040] Step 4.1: Multimodal weighted data The training set and the test set are divided into a training set and a test set according to a certain ratio, and a fully connected neural network is used to train the training set. In the fully connected neural network, two hidden layers are set. The fully connected neural network automatically learns and extracts input weight modal data through multi-layer neuron connections. The features in the weighted modal data are interpolated through training ;

[0041] Step 4.2: Pass Function, processing weighted mode vector:

[0042]

[0043] in, refers to the parameters obtained through training, is the loss calculation function, is the weighted modal data The number of modes in the local feature dataset is minimized by the above method value.

[0044] A further improvement of the present invention is that in step 5, the client training result is sent to the server, and the aggregation operation specifically includes the following steps:

[0045] Step 5.1: After receiving the model parameter data sent by the client, the server performs an aggregation operation. Using the FedAvg framework, the model parameters uploaded by the client are aggregated to obtain the local model parameter updates of each client device. After the federated average, the global model parameters for the next round of training are obtained. Specifically:

[0046]

[0047] in, Indicates Wheel equipment The local model parameter update, It is in The set of client devices participating in the training in the round, Yes Equipment The amount of local data, is the total data volume of all participating devices, that is , Indicates The server obtains the global model parameters of the round after completing the weighted average;

[0048] Step 5.2: Send the global model parameters to the client participants for updating and then repeat the iteration.

[0049] The beneficial effects of the present invention are:

[0050] The present invention constructs a feature extraction module and a modality prediction module, and utilizes multi-layer perceptron technology to make a relatively accurate prediction of the missing modality through the existing modality.

[0051] The present invention extracts feature information of multimodal data, constructs feature labels and modal weights, marks and processes the information of each modality, and fuses all modalities into a complete multimodal feature vector through a multi-head attention mechanism, effectively filtering out redundant feature dimensions in each modality, and combining weights to better utilize the information of each modality, thereby obtaining modal feature information that is more suitable for federated learning processing.

[0052] The present invention combines the FedAvg framework to weightedly aggregate the multimodal training data of multiple clients on a central server when dealing with the multimodal missing problem, thereby improving the overall model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of the multimodal federated learning method of the present invention.

[0054] Figure 2 It is a workflow diagram of the present invention for predicting missing modalities.

[0055] Figure 3 It is a flow chart of the modality fusion process of the present invention.

[0056] Figure 4 It is a workflow diagram of the present invention for performing federated aggregation.

[0057] Figure 5 This is a comparison chart between the federated learning method of the present invention and the existing algorithm under different numbers of clients.

[0058] Figure 6 It is a comparison chart of the method of the present invention and the existing algorithm under different modal missing rates.

[0059] Figure 7 It is a comparison chart of the method of the present invention and the existing algorithm under different modal missing rates. DETAILED DESCRIPTION

[0060] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0061] Figure 1 It is a method flow chart of an example of the present invention. For the initial multimodal data, firstly, the modal features with strong correlation with the missing modality are screened out, and these features are trained by a multi-layer perceptron to complete the prediction of the missing modality and obtain the aligned modal information. For the aligned modal information, the present invention tags the features in each modality based on the importance of the feature to the modality, and combines the feature information with the feature label to form a feature vector. According to the importance of the modality to the training model, the modal label is designed, and the context vector and the modal information are combined to form a single modal vector. The single modal vectors are spliced ​​to obtain a multimodal vector. The obtained multimodal vector is trained on the client side through a fully connected neural network, and the trained training model is uploaded to the server side. After aggregating the models uploaded by each client, the global model is forwarded to the client side for the next round of training.

[0062] like Figure 1 As shown, the present invention is a robust multimodal federated learning method, which obtains high-precision training results in the absence of modal data, and specifically includes the following steps:

[0063] Step 1: For clients with missing modalities, predict the missing modal data based on the existing modal data: first, filter out features that are highly correlated with the missing modal data from the existing modal data, use a multi-layer perceptron to train this part of the feature data, predict the unknown modality, and complete the modal alignment operation.

[0064] Figure 2 This is the workflow diagram of the present invention for predicting missing modalities. The process of predicting missing modalities is divided into a feature processing module and a modality prediction module. The feature processing module is responsible for extracting features that are highly correlated with the missing modalities from known modalities. For these features, determine whether they are suitable for predicting missing modalities. The features that can be used to predict missing modalities are placed separately in a feature queue for predicting missing modalities, and unavailable modal features are discarded. The modality prediction module is responsible for evaluating the importance of the target modality based on the features in the feature queue, and obtaining feature weights for predicting missing modalities. Based on the feature weights, a weighted feature queue for predicting missing modalities is obtained, and the data in the feature queue is normalized. Through multi-layer perceptron training, the predicted missing modality is obtained, which is interpolated into the missing modal information to complete the modality information alignment.

[0065] The specific steps include:

[0066] Step 1.1, feature extraction: extract features that are highly correlated with the missing modal data from the existing modal data, and determine whether the feature information can be used to predict the missing modality. If the feature information is available, it is put into the feature queue for predicting the missing modality. Specifically:

[0067] Step 1.1.1: By exploring the relationship between the features of the existing modal data and the missing modal data, the features with strong correlation with the missing modal data are extracted. The feature extraction process It is expressed as:

[0068]

[0069] in, Indicates the current mode data, Indicates that from the existing mode The features extracted from the target are used to predict the feature information of the target missing modality;

[0070] Step 1.1.2: For the extracted features, filter out the features that cannot be used for the prediction of missing modal data; if some features have high noise or lack sufficient discrimination, discard them;

[0071] Step 1.1.3: Put the extracted features that can be used for missing modal data prediction into a feature queue for predicting missing modal data for missing modal data prediction.

[0072] Step 1.2, modality prediction: score and normalize the importance of the target missing modality according to the features in the feature queue to obtain the rough target modality data After training with a multi-layer perceptron, we get a more realistic predicted missing mode, specifically:

[0073]

[0074] in, Represents the characteristics The rating, It means that after the filtered feature information obtains the weight, all weighted feature information is merged. LN represents the normalization function, which is used to prevent the problem of gradient explosion or disappearance. The coarse target modal data is obtained by calculating the weighted feature information of the target missing modality in the existing modality. , for the obtained coarse target modal data After merging, two rounds of multi-layer perceptron calculations allow the model to better capture the nonlinear relationship between input features, improve the quality of feature expression and the overall performance of the model, and obtain the final predicted missing mode. , the interpolation operation is completed.

[0075] Step 2: Process the features in the existing mode and the predicted missing mode: For each feature, label it according to the importance of its influence on the feature, and fuse the features with the feature labels to form a feature vector. The specific steps include:

[0076] Step 2.1: Extract the modal feature data of each modality from the existing modality and the predicted missing modality, such as sound, image, etc. For each modality in the existing modality and the predicted missing modality The feature samples included , according to the importance of feature samples, construct feature labels and normalize them, specifically:

[0077] in, represents the calculated feature label, Represents the specific calculation process of training the encoding network, which is used to determine the feature sample to feature label The importance of training, refer to Activation function maps data to the range of 0 to 1, which is more suitable for learning complex feature information;

[0078] Step 2.2: For each mode The features of the modal feature vector are synthesized and normalized weighted feature information is concatenated to obtain a one-dimensional modal feature vector. :

[0079]

[0080] in, represents the weighting process of feature information, and the accumulation process represents the weighting of each modality. In Weighted feature information concatenation.

[0081] Figure 3 is a flow chart of the modal fusion process of the present invention. First, according to the importance of the modality in the model training, the modal weight matrix is ​​set, and the set modal weight is combined with the modal data to obtain the original weighted modal matrix. After the weighted modal matrix is ​​added with the bias, it is placed in the multi-layer perceptron for training to obtain a multimodal data matrix for obtaining attention weights. Then, by setting the context vector, it is combined with the multimodal data matrix for obtaining attention weights, and used The function is normalized to obtain the attention weight matrix. Finally, after each modal data obtains its attention weight independently, multiple unimodal data with attention weights are obtained. These unimodal data with attention weights are concatenated to obtain the multimodal data used for training operations.

[0082] The modal fusion synthesis modal vector specifically includes the following steps:

[0083] Step 3.1: Process the multimodal data. After obtaining the weights, add the bias and train it through a multilayer perceptron to obtain preprocessed multimodal data for further obtaining importance scores:

[0084]

[0085] in, The modal eigenvector After splicing, the multimodal data is the weight matrix, is the bias, and the spliced ​​multimodal data is transformed through a layer of multi-layer perceptron Converting to multimodal data ;

[0086] Step 3.2: By setting the context vector and performing normalization, the importance score of the final description modal information is obtained:

[0087]

[0088] in, is the context vector. By setting different values ​​for it, a multi-head attention mechanism is implemented to dynamically adjust the weight of the modal data. And processed multimodal data After combining the transposed matrices of Function gets normalized importance score , represents the weight of each modality, that is, the importance of the modality in the final representation, The accumulated matrix is ​​the modality label matrix;

[0089] Step 3.3: All modal eigenvectors and importance rating Combine and concatenate to obtain multimodal weighted data for training:

[0090]

[0091] in, Representing modality The importance rating of Modal The modal information of Represents all modal eigenvectors Combined with the importance score and concatenated to obtain the final multimodal weighted data .

[0092] Step 4: Locally train the fused multimodal weighted data: Use the redundant computing resources of the client to train the multimodal weighted data, relieve the computing pressure of the central server, and avoid the leakage of client data. Each client uses a fully connected neural network model locally to train the multimodal weighted data to obtain a multimodal model. A fully connected neural network is established locally for the client. The process is as follows:

[0093] Step 4.1: Multimodal weighted data The training set and the test set are divided into a training set and a test set according to a ratio of 9:1. The training set is trained using a fully connected neural network. In the fully connected neural network, two hidden layers are set to avoid overfitting that may be caused by too many hidden layers, so as to better process the abstract initial data. The fully connected neural network automatically learns and extracts input weight modal data through multi-layer neuron connections. The features in the weighted modal data are interpolated through training ; Help the model better understand the data to improve the model's ability to represent the data.

[0094] Step 4.2: Pass Function, processing weighted mode vector:

[0095]

[0096] in, refers to the parameters obtained through training, is the loss calculation function, is the weighted modal data The number of modes in the local feature dataset is minimized by the above method value.

[0097] Step 5: Upload the multimodal model trained locally on each client obtained in step 4 to the server, share the training results of each client on the server, realize collective learning and optimization of the multimodal model, adopt the corresponding federated aggregation algorithm to perform aggregation operations according to actual conditions, update the global multimodal model, and redistribute the aggregated multimodal model to each client for the next round of iteration until the multimodal model converges.

[0098] Figure 4 This is the workflow diagram of the federal aggregation of the present invention. Each client device uses local multimodal data to train the model and update the local model parameters. After the client completes the training locally, it uploads the trained model to the server. This step only uploads the model parameters, not the local data, thereby protecting the privacy of the user. After the central server receives the model parameters uploaded by all client devices, it performs a weighted average on these parameters based on the amount of data of each device. After weighted averaging, the central server updates the parameters of the global model and forwards the updated global model to each client device for the next round of training. The specific steps of performing the aggregation operation include the following:

[0099] Step 5.1: After receiving the model parameter data sent by the client, the server performs an aggregation operation. Using the FedAvg framework, the model parameters uploaded by the client are aggregated to obtain the local model parameter updates of each client device. After the federated average, the global model parameters for the next round of training are obtained. Specifically:

[0100]

[0101] in, Indicates Wheel equipment The local model parameter update, It is in The set of client devices participating in the training in the round, Yes Equipment The amount of local data, is the total data volume of all participating devices, that is, , Indicates The server obtains the global model parameters of the round after completing the weighted average;

[0102] Step 5.2: Send the global model parameters to the client participants for updating and then repeat the iteration.

[0103] like Figure 5 The comparison diagram of the method of the present invention and the existing algorithm under different numbers of clients is shown in the attached figure. Figure 5 It can be seen that the method of the present invention performs well in the multi-client federated learning scenario, especially maintaining a high and stable accuracy rate when the number of clients increases, indicating that it has better robustness in handling multi-client environments.

[0104] like Figure 6 The comparison diagram of the method of the present invention and the existing algorithms under different modality missing rates shows that when the modality missing rate is low, the method of the present invention and other existing algorithms can maintain a high accuracy rate. However, when the modality missing rate is high, the method of the present invention can still maintain a high accuracy rate and is the most robust when processing missing modality data.

[0105] like Figure 7 The performance comparison of the method of the present invention and the existing algorithm on the CMU-MOSI dataset is shown. According to the attached figure, it can be seen that when they process the same sentiment classification task, the different sentiment categories of the present invention have less overlap and the intervals between different categories are more obvious, which is beneficial to sentiment classification. The present invention can produce better training results when training in the case of modality loss.

[0106] The present invention effectively solves the problem of reduced training model accuracy when the modality is missing by predicting the missing modality and fusing it with multimodal data, reduces the impact of modality missing on training accuracy, and improves the reliability of federated learning.

Claims

1. A robust multimodal federated learning method, characterized by: The multimodal federated learning method obtains high-precision training results in the absence of modal data. The modal data is text, speech, image or video. Specifically, the method includes the following steps: Step 1: For clients with missing modalities, predict the missing modal data based on the existing modal data: first, select features with a strong correlation with the missing modal data from the existing modal data, use a multi-layer perceptron to train this part of the feature data, predict the unknown modality, and complete the modal alignment operation; Step 2: Process the features in the existing mode and the predicted missing mode: for each feature, label it according to the importance of the feature to its location, and fuse the features with the feature labels to form a feature vector; Step 3: Fuse the modal data aligned in step 1: design a modal label matrix based on whether the modal data is predicted data and the importance of the modal data, fuse the feature vector constructed in step 2 with the modal label in the modal label matrix, obtain multi-modal weighted data, and synthesize the modal vector of each client; Step 4: Locally train the fused multimodal weighted data: Use redundant computing resources of the client to train the multimodal weighted data. Each client locally uses a fully connected neural network model to perform training operations on the multimodal weighted data to obtain a multimodal model. Step 5: upload the local training multimodal model of each client obtained in step 4 to the server, share the training results of each client on the server, realize the collective learning and optimization of the multimodal model, adopt the corresponding federated aggregation algorithm to perform aggregation operation according to the actual situation, update the global multimodal model, and redistribute the aggregated multimodal model to each client for the next round of iteration until the multimodal model converges; Wherein: the step 2 specifically comprises the following steps: Step 2.1: Extract the modal feature data of each modality from the existing modality and the predicted missing modality. For each feature sample f contained in each modality M in the existing modality and the predicted missing modality, construct a feature label according to the importance of the feature sample and normalize it. Specifically, in, represents the calculated feature label, Represents the specific calculation process of training the encoding network, which is used to determine the feature sample to feature label The importance of training, sigmoid() refers to the sigmoid activation function, which maps the data to the range of 0 to 1; Step 2.2: For each mode M feature, synthesize the normalized weighted feature information and perform feature concatenation to obtain a one-dimensional modal feature vector X M : in, It represents the weighting process of feature information, and the accumulation process represents the concatenation of f weighted feature information in each modality M; In step 3, the modal fusion and synthesis of the modal vector specifically include the following steps: Step 3.1: Process the multimodal data. After obtaining the weights, add the bias and train it through a multilayer perceptron to obtain preprocessed multimodal data for further obtaining importance scores: w=mlp(WX+b) Where X is the modal eigenvector X M The spliced ​​multimodal data, W is the weight matrix, b is the bias, and the spliced ​​multimodal data X is converted into multimodal data w through a layer of multi-layer perceptron; Step 3.2: By setting the context vector and performing normalization, the importance score of the final description modal information is obtained: r=softmax(w T c) Among them, c is the context vector. By setting different values ​​for it, a multi-head attention mechanism is implemented to dynamically adjust the weight of the modal data. After combining the context vector c and the transposed matrix of the processed multimodal data w, the normalized importance score r is obtained through the softmax function, which represents the weight of each modality, that is, the importance of the modality in the final representation. i The accumulated matrix is ​​the modality label matrix; Step 3.3: All modal eigenvectors X M Combined and concatenated with the importance score r, we obtain multimodal weighted data for training: Among them, r i represents the importance score of mode i, X i modal information of mode i, Represents all modal eigenvectors X M The final multimodal weighted data H is obtained by combining and concatenating the importance scores.

2. A robust multimodal federated learning method according to claim 1, characterized in that: In the step 1, the missing modal data is predicted based on the existing modal data, which specifically includes the following steps: Step 1.1, feature extraction: extract features that are highly correlated with the missing modal data from the existing modal data, determine whether the feature information can be used to predict the missing modality, and if the feature information is available, put it into the feature queue for predicting the missing modality; Step 1.2, modality prediction: score and normalize the importance of the target missing modality according to the features in the feature queue to obtain the rough target modality data After multi-layer perceptron training, the predicted missing mode is obtained, specifically: in, Represents the characteristics The rating, It means that after the filtered feature information is weighted, all weighted feature information is merged. LN represents the normalization function. The coarse target modal data is obtained by calculating the weighted feature information of the target missing modality in the existing modality. The obtained coarse target modal data After merging, two rounds of multi-layer perceptron calculations are performed to obtain the final predicted missing mode z m , the interpolation operation is completed.

3. A robust multimodal federated learning method according to claim 2, characterized in that: The step 1.1 feature extraction specifically includes the following steps: Step 1.1.1: By exploring the relationship between the features of the existing modal data and the missing modal data, the features with strong correlation with the missing modal data are extracted. The feature extraction process f of the existing modal m is m It is expressed as: Among them, data m represents the data of the existing mode m, represents the features extracted from the existing modality m; Step 1.1.2: For the extracted features, filter out the features that cannot be used for the prediction of missing modal data and discard them; Step 1.1.3: Put the extracted features that can be used for missing modal data prediction into a feature queue for predicting missing modal data for missing modal data prediction.

4. The robust multimodal federated learning method according to claim 1, characterized in that: In step 4, a fully connected neural network is established locally on the client, and the process is as follows: Step 4.1, dividing the multimodal weighted data H into a training set and a test set, using a fully connected neural network to perform a training operation on the training set, in which two hidden layers are set, and the fully connected neural network is connected through multiple layers of neurons to automatically learn and extract features in the input weighted modal data H, and the weighted modal data H is completed by training interpolation; Step 4.2: Process the weighted mode vector through the Loss function: Among them, ω refers to the parameters obtained through training, Loss is the loss calculation function, and N is the number of modes in the weighted modal data H. The Loss value of the local feature data set is minimized in the above way.

5. The robust multimodal federated learning method according to claim 1, characterized in that: In step 5, the client training results are sent to the server, and the aggregation operation specifically includes the following steps: Step 5.1: After receiving the model parameter data sent by the client, the server performs an aggregation operation. Using the FedAvg framework, the model parameters uploaded by the client are aggregated to obtain the local model parameter updates of each client device. After the federated average, the global model parameters for the next round of training are obtained. Specifically: in, represents the local model parameter update of device k in round t, S t is the set of client devices participating in training in round t, n k is the local data volume of device k, and n is the total data volume of all participating devices, that is, w t+1 Represents the global model parameters of the t+1th round. After completing the weighted average, the server obtains the global model parameters; Step 5.2: Send the global model parameters to the client participants for updating and then repeat the iteration.

Images