A multimodal federated learning method and related equipment for compensating for modality loss

Abstract

The present application relates to the technical field of multimodal data, and provides a multimodal federated learning method and related equipment for compensating for modality loss, the method comprising: determining a missing client from all target clients; extracting features of the modal data of the missing client to obtain data features, fusing all data features to obtain final fused features, extracting time features of the modal data of the missing client, fusing all time features to obtain final time features, and obtaining data of the missing modality based on the final fused features and the final time features; taking the target client and the missing client after obtaining the data of the missing modality as the final client, training each modal model using the data of all modalities of the final client to obtain a trained modal model; integrating all trained modal models to obtain a multimodal model. The method of the present application can improve the quality of the multimodal model.

Description

Technical Field

[0001] The present application relates to the field of multimodal data technology, and in particular to a multimodal federated learning method and related equipment for compensating for modality loss. Background Art

[0002] Today, the development of multimodal technology enables computers to acquire, analyze, and understand information from multiple sensory modalities, providing more comprehensive and accurate solutions for applications across various fields. This has also driven innovation in areas such as human-computer interaction, sentiment analysis, visual speech recognition, and healthcare. However, multimodal technology requires extensive data training, which can lead to device overload and pose the risk of privacy leaks during data transmission. This is where the emergence of multimodal technology based on the federated learning framework comes in.

[0003] Federated learning completes model training only on the client. The server receives the model sent by the client and, after model aggregation, sends the multimodal global model to the client for the next round of training. This effectively alleviates the problem of device overload caused by large amounts of data, and the data does not leave the local machine, effectively protecting privacy and preventing the risk of data leakage.

[0004] However, when multimodal technology is involved, some clients are missing modalities. This prevents these clients from iteratively updating the full multimodal global model delivered by the server. This results in fewer clients training the model, fewer models uploaded to the server, and low-quality multimodal models generated by model aggregation. This shows that existing multimodal federated learning methods suffer from the problem of low-quality multimodal models. Summary of the Invention

[0005] This application provides a multimodal federated learning method and related equipment to make up for modality loss, which can solve the problem of low quality of the generated multimodal model.

[0006] In a first aspect, an embodiment of the present application provides a multimodal federated learning method for compensating for modality loss, the multimodal federated learning method comprising:

[0007] Get the data of all modalities stored by each target client;

[0008] According to the data of all modalities stored in each target client, the missing clients having modality missing data are determined from all target clients;

[0009] For each missing client, feature extraction is performed on the data of each modality of the missing client to obtain multiple data features. All data features are fused to obtain the final fused feature. The time feature of the data of each modality of the missing client is extracted and all time features are fused to obtain the final time feature. Based on the final fused feature and the final time feature, the data of the missing modality of the missing client is obtained.

[0010] For each modality, obtain the modal model corresponding to the modality, take each target client and each missing client after obtaining the data of the missing modality as the final client, use the data of all modalities of each final client to train the modal model corresponding to the modality, and obtain the trained modal model of each final client;

[0011] All trained modal models are integrated to obtain a multimodal model.

[0012] Optionally, feature extraction is performed on the data of each modality of the missing client to obtain multiple data features, including:

[0013] By formula:

[0014]

[0015] Get the Data characteristics of the data of each modality ;

[0016] in, , Indicates the number of modalities with missing clients, Indicates the The data features of the first dimension of the modal data, Indicates the The data features of the second dimension of the modal data, Indicates the The first modality of data Data features of dimensions, express A collection of data features in multiple dimensions.

[0017] Optionally, all data features are fused to obtain the final fused features, including:

[0018] Calculate the attention weight between each two modal data, and based on all attention weights and all data features, calculate multiple modal vectors corresponding to the data of each modality;

[0019] All modal vectors are fused to obtain the final fusion feature.

[0020] Optionally, based on all attention weights and all data features, multiple modality vectors corresponding to the data of each modality are calculated, including:

[0021] By formula:

[0022]

[0023] Calculate the The interaction of data from different modalities The modal vector of the data of each mode;

[0024] in, Indicates the The data of each modality are The attention weight of the data of each modality, Indicates the The data characteristics of the modal data, , , Indicates the number of modalities with missing clients.

[0025] Optionally, all modal vectors are fused to obtain the final fused features, including:

[0026] By formula:

[0027]

[0028]

[0029] Calculate the final fusion features ;

[0030] in, represents the fused modal vector of the data of the first modality, represents the fused modal vector of the data of the second modality, Indicates the The fused modal vector of the modal data, Indicates the The fused modal vector of the modal data, Indicates the interaction of the data in the first mode The modal vector of the data of the modes, Indicates in The interaction of data from different modalities The modal vector of the data of the modes, represents the Hadamard product.

[0031] Optionally, the modal model corresponding to the modality is trained using data of all modalities of each final client to obtain a trained modal model corresponding to the modality of each final client, including:

[0032] For each end client, perform the following steps:

[0033] The target data of the same modality as the modal model corresponding to the modality is determined from the data of all modalities of the final client, the trained parameters of the modal model are calculated using the target data, and the trained parameters are substituted into the modal model to obtain the trained modal model of the final client.

[0034] Optionally, use the target data to calculate the trained parameters of the modal model, including:

[0035] By formula:

[0036]

[0037] Calculate the The final client corresponds to Parameters of the modal model after training ;

[0038] in, Indicates the The final client and Target data of the same mode of the modal model, Indicates the The initial parameters of the modal model, , Indicates the number of final clients, , represents the total number of modes, represents the learning rate, represents the number of training rounds, Represents a training operation.

[0039] Optionally, all trained modal models are integrated to obtain a multimodal model, including:

[0040] By formula:

[0041]

[0042]

[0043] Calculating multimodal parameters ;

[0044] in, Indicates the The fusion weight of the modal model, Indicates the The fusion parameters of the modal model.

[0045] In a second aspect, an embodiment of the present application provides a multimodal federated learning device that compensates for modality loss, including:

[0046] The acquisition module obtains the data of all modalities stored in each target client;

[0047] A determination module, which determines missing clients whose data has modal missing from all target clients based on the data of all modalities stored in each target client;

[0048] The feature extraction module extracts features from the data of each modality of each missing client, obtains multiple data features, fuses all data features to obtain the final fused features, extracts the time features of the data of each modality of the missing client, fuses all time features to obtain the final time features, and obtains the data of the missing modality of the missing client based on the final fused features and the final time features;

[0049] The training module obtains the modal model corresponding to each modality for each modality, takes each target client and each missing client after obtaining the data of the missing modality as the final client, and uses the data of all modalities of each final client to train the modal model corresponding to the modality, thereby obtaining the trained modal model of each final client;

[0050] The integration module integrates all trained modal models to obtain a multimodal model.

[0051] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multimodal federated learning method for compensating for modality loss is implemented.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multimodal federated learning method for compensating for modality loss.

[0053] The above solution of the present application has the following beneficial effects:

[0054] In an embodiment of the present application, data of all modalities stored in each target client are obtained, and then, based on the data of all modalities stored in each target client, missing clients with modal missing data are determined from all target clients, and then, for each missing client, feature extraction is performed on the data of each modality of the missing client to obtain multiple data features, all data features are fused to obtain a final fused feature, and time features of the data of each modality are extracted, all time features are fused to obtain a final time feature, and based on the final fused feature and the final time feature, the data of the missing modality of the missing client is obtained, and then, for each modality, the modal model corresponding to the modality is obtained, and each target client and each missing client after obtaining the data of the missing modality are used as final clients, and the modal model corresponding to the modality is trained using the data of all modalities of each final client to obtain the trained modal model of each final client, and finally, all trained modal models are integrated to obtain a multimodal model. Among them, the final fusion features and final time features of the missing client are used to obtain the data of the missing modality, taking into account both the data itself and the time series of the data, thereby improving the accuracy of the data of the missing modality and reducing the situation of missing client data. It avoids the situation where the client with missing modality cannot use the model downloaded from the server and thus cannot perform model training. After the client's modal data is completed, the client can also perform subsequent model training like other clients with complete data, thereby improving the quality of the multimodal model.

[0055] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0057] Figure 1 A flowchart of a multimodal federated learning method for compensating for modality loss provided in one embodiment of the present application;

[0058] Figure 2 A schematic diagram of a process for generating modal data according to an embodiment of the present application;

[0059] Figure 3 A schematic diagram of the structure of a multimodal model generation system for compensating for modality loss provided in one embodiment of the present application;

[0060] Figure 4A schematic diagram of the structure of a multimodal federated learning device for compensating for modality loss provided in one embodiment of the present application;

[0061] Figure 5 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0062] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0063] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0064] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0065] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0066] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0067] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0068] In response to the problem that existing clients have difficulty training models, an embodiment of the present application provides a multimodal federated learning method to compensate for modal missingness. The multimodal federated learning method obtains data of all modalities stored by each target client, and then determines the missing clients whose data has modal missingness from all target clients based on the data of all modalities stored by each target client. Then, for each missing client, feature extraction is performed on the data of each modality of the missing client to obtain multiple data features, all data features are fused to obtain the final fused feature, and the time feature of the data of each modality is extracted, all time features are fused to obtain the final time feature. Based on the final fused feature and the final time feature, the data of the missing modality of the missing client is obtained, and then the modal model corresponding to the modality is obtained for each modality. Each target client and each missing client after obtaining the data of the missing modality are regarded as the final client, and the modal model corresponding to the modality is trained using the data of all modalities of each final client to obtain the trained modal model of each final client. Finally, all trained modal models are integrated to obtain a multimodal model. Among them, the final fusion features and final time features of the missing client are used to obtain the data of the missing modality, taking into account both the data itself and the time series of the data, thereby improving the accuracy of the data of the missing modality and reducing the situation of missing client data. It avoids the situation where the client with missing modality cannot use the model downloaded from the server and thus cannot perform model training. After the client's modal data is completed, the client can also perform subsequent model training like other clients with complete data, thereby improving the quality of the multimodal model.

[0069] Next, an example description is given of the multimodal federated learning method provided in this application to compensate for modality loss.

[0070] like Figure 1 As shown, the multimodal federated learning method provided by this application to compensate for modality loss includes the following steps:

[0071] Step 11: Obtain data of all modalities stored in each target client.

[0072] The data of all modalities mentioned above includes data for each modality of the target client. For example, the data of the first modality may be text data, the data of the second modality may be audio data, and the data of the third modality may be image data. Each modality's data may include multiple data items, such as image data including 10 images acquired at different time points. The target client mentioned above is the client participating in multimodal federated learning.

[0073] In some embodiments of the present application, data of all modalities stored in the target client may be obtained by accessing the data storage device of the target client.

[0074] Step 12: Based on the data of all modalities stored in each target client, determine the missing clients whose data are missing in a modality from all target clients.

[0075] It should be noted that, due to the different local data situations of the target client, when acquiring data of all modalities, it may happen that the target client does not have data corresponding to the modality.

[0076] Specifically, according to the multiple modalities that need to participate in the generation of the multimodal model set by the federated learning server, if at least one of the above modalities is missing in the data of all modalities of the target client, the target client is regarded as a missing client.

[0077] For example, among three target clients, the first target client and the second target client both have data in three modalities: text data, audio data, and image data, while the third target client only has text data and audio data but no image data, and this target client is considered to be a missing client.

[0078] In step 13, for each missing client, feature extraction is performed on the data of each modality of the missing client to obtain multiple data features, all data features are fused to obtain the final fused feature, and the time feature of the data of each modality of the missing client is extracted, all time features are fused to obtain the final time feature, and based on the final fused feature and the final time feature, the data of the missing modality of the missing client is obtained.

[0079] In some embodiments of the present application, the above-mentioned steps of extracting features from the data of each modality of the missing client to obtain multiple data features, fusing all the data features to obtain a final fused feature, extracting the time features of the data of each modality of the missing client, fusing all the time features to obtain a final time feature, and obtaining the data of the missing modality of the missing client based on the final fused feature and the final time feature are specifically as follows:

[0080] In the first step, feature extraction is performed on the data of each modality of the missing client to obtain multiple data features.

[0081] Specifically, through the formula:

[0082]

[0083] Get the Data characteristics of the data of each modality ;

[0084] in, , Indicates the number of modalities with missing clients, Indicates the The data features of the first dimension of the modal data, Indicates the The data features of the second dimension of the modal data, Indicates the The first modality of data Data features of dimensions, express A collection of data features in multiple dimensions.

[0085] It should be noted that since the dimensions of different data features may be different, after obtaining the data features, the attention mechanism can be used to filter the importance of each dimension by querying the interactivity between the dimensions within the data features, and select data features of multiple important dimensions to enter the next step of processing. The number of important dimensions corresponding to all data features is the same to facilitate subsequent calculations.

[0086] In the second step, all data features are fused to obtain the final fused features.

[0087] First, the attention weights between the data of each two modalities are calculated, and based on all attention weights and all data features, multiple modal vectors corresponding to the data of each modality are calculated.

[0088] Specifically, through the formula:

[0089]

[0090] Calculate the The interaction of data from different modalities The modal vector of the data for each mode.

[0091] in, Indicates the The data of each modality are The attention weight of the data of each modality, Indicates the The data characteristics of the modal data, , , Indicates the number of modalities with missing clients.

[0092] Then, all modal vectors are fused to obtain the final fused features.

[0093] Specifically, through the formula:

[0094]

[0095]

[0096] Calculate the final fusion features .

[0097] in, represents the fused modal vector of the data of the first modality, represents the fused modal vector of the data of the second modality, Indicates the The fused modal vector of the modal data, Indicates the The fused modal vector of the modal data, Indicates the interaction of the data in the first mode The modal vector of the data of the modes, Indicates in The interaction of data from different modalities The modal vector of the data of the modes, represents the Hadamard product.

[0098] It should be noted that after obtaining the final fusion feature, it is necessary to perform normalization, activation function, dimensionality reduction and other operations on the final fusion feature so that the data form of the final fusion feature meets the data form of subsequent steps.

[0099] The third step is to extract the time features of the data of each modality of the missing client, fuse all the time features, and obtain the final time features.

[0100] In some embodiments of the present application, local sensitive hashing is first used to preprocess the modal data to screen out key features or data points, and then a long short-term memory network (LSTM) is used to extract the time features of the preprocessed modal data. Finally, all time features can be fused through calculations such as the Hadamard product to obtain the final time features.

[0101] The fourth step is to obtain the data of the missing modality of the missing client based on the final fusion features and the final time features.

[0102] In some embodiments of the present application, the final fused features and final time features can be processed using an inverse diffusion model to obtain the data of the missing modality of the missing client. Specifically, the final fused features and final time features are used as the input of the last time step of the inverse diffusion model. The inverse diffusion model gradually generates the output of the previous time step until the output of the 0th time step is generated. The output of the 0th time step is the data of the missing modality.

[0103] It should be noted that the expression for processing the final fusion feature and the final time feature in the inverse diffusion model is:

[0104]

[0105]

[0106] in, Indicates that from The joint probability distribution from time steps to time step 0, Indicates that from The main mode from the time step to the 0th time step (i.e. the final fusion feature), Indicates that from The auxiliary mode from the time step to the 0th time step (i.e. the final time feature), Indicates the The probability distribution of the main mode in each time step, Indicates the The joint probability distribution of time steps, , represents the last time step, Indicates the The main mode of the time step, Indicates the The main mode of the time step, Indicates the Auxiliary modes of time steps, Indicates in time steps, the parameters The mean of the parameterized neural network predictions.

[0107] For example, if a missing client has text and audio data but lacks image data, the above steps are used to obtain the final fused features and final time features based on the text and audio data. Based on these final fused features and final time features, the above-described inverse diffusion model is used to obtain image data. When the inverse diffusion model processes the final fused features and final time features to generate data for other modalities, it can learn the joint distribution between different data modalities. This allows the model to learn not only the data distribution within each modality but also the mapping relationship between different modalities. At each step of the inverse diffusion process, the model predicts the next state based on the current state and given conditions (i.e., the extracted features). Each step relies on the current modality to guide the transformation from random noise to the final modality (the desired missing modality).

[0108] It is worth mentioning that by using the final fusion features and final time features of the missing client to obtain the data of the missing modality, both the data itself and the time series of the data are taken into consideration, which improves the accuracy of the data of the missing modality and reduces the situation of missing client data.

[0109] The above-mentioned inverse diffusion model is exemplified below with reference to a specific example.

[0110] like Figure 2 As shown, the composite main mode (i.e. the final fusion feature mentioned above) is used as the input of the last time step of the inverse diffusion model , the time series auxiliary information (i.e. the final time feature mentioned above) is used as the auxiliary input of the last time step , Indicates gate control, Represents the activation function. In this embodiment, only when the conditional threshold is reached in each time step (for example, the modal model is a model for generating a person's walking, the movement trajectory of the person's legs is the main mode, and the weather is auxiliary information. On rainy days, people's feet will be lifted lower, so the conditional threshold can be set to whether the weather is rainy. The auxiliary information has only two types of data, label 1 for raining and 0 for not raining. When the step length is t, the auxiliary modal label received is 1, and the auxiliary information is added), the gate is turned on, the activation function is activated, and the time series auxiliary information can be substituted into the generation of the composite main mode. After multiple time steps, the inverse diffusion model generates the first Composite main mode of time steps And generate the Time series auxiliary information of time steps , and according to The composite main mode of the time step and the The time series auxiliary information of the time step is generated Composite main mode of time steps And generate the Time series auxiliary information of time steps , after multiple time steps, the composite main mode of the first time step is generated And generate the time series auxiliary information of the first time step , and finally according to the composite main mode of the first time step And generate the time series auxiliary information of the first time step Generate the composite main mode at time step 0 , that is, the data of the missing modality.

[0111] In step 14, for each modality, the modal model corresponding to the modality is obtained. Each target client and each missing client after obtaining the data of the missing modality are regarded as the final client. The modal model corresponding to the modality is trained using the data of all modalities of each final client to obtain the trained modal model of each final client.

[0112] It should be noted that all modal models are of the same type (e.g., prediction model, evaluation model, etc.) and share the same structure and expression; only the input data and parameters differ. For each end client, each modality corresponds to a modal model. After training all modal models using the data from all modalities of that end client, a trained modal model corresponding to each modal model is obtained.

[0113] In some embodiments of the present application, the modal model may be a modal model constructed by the server corresponding to all target clients after receiving modal data from a random target client, and then the server sends all constructed modal models to each final client. The steps of training the modal model corresponding to the modality using the data of all modalities of each final client to obtain the trained modal model for each final client are specifically as follows:

[0114] For each end client, perform the following steps:

[0115] The target data of the same modality as the modal model is determined from the data of all modalities of the final client, the trained parameters of the modal model are calculated using the target data, and the trained parameters are substituted into the modal model to obtain the trained modal model of the final client.

[0116] Specifically, through the formula:

[0117]

[0118] Calculate the The final client corresponds to Parameters of the modal model after training .

[0119] in, Indicates the The final client and Target data of the same mode of the modal model, Indicates the The initial parameters of the modal model, , Indicates the number of final clients, , represents the total number of modes, represents the learning rate, represents the number of training rounds, Represents a training operation.

[0120] Exemplarily, the above-mentioned training operation can be an operation process of model training such as a loss function training method and an objective function training method. The above steps are all performed on the final client. Each final client uses its own data to train the corresponding modal model. If there are 3 modal models, each final client needs to train 3 modal models to obtain 3 trained modal models.

[0121] It is worth mentioning that by making up for the missing client data, each client can use its own data to train the modal model.

[0122] In step 15, all trained modal models are integrated to obtain a multimodal model.

[0123] Specifically, through the formula:

[0124]

[0125]

[0126] Calculating multimodal parameters .

[0127] in, Indicates the The fusion weight of the modal model, Indicates the The fusion parameters of the modal model.

[0128] In some embodiments of the present application, this step can be performed on the server corresponding to all target clients. That is, after the final client obtains the trained modal model, the trained modal model is uploaded to the server, and the server calculates the multimodal parameters. If the performance of the multimodal model does not meet expectations, the process can return to step 14, and each final client can be instructed to train the modal model again and upload the trained modal model to the server until the performance of the multimodal model meets expectations.

[0129] It should be noted that when executing this step, according to the above formula for calculating the multimodal parameters, it is necessary to first calculate the fusion parameters of the modal model based on the parameters of all trained modal models corresponding to each modality, and then fuse all the fusion parameters to obtain the multimodal parameters.

[0130] For example, when the server sends the modal model to the final client, it can send data of the standard modality of each modality that meets the preset quality standards to the final client. After receiving the data of the standard modality, the final client calculates the cosine similarity between the data of the standard modality and the data of its own modality. If the cosine similarity is less than the similarity threshold, the final client is marked as a poor-performing client. In addition, after receiving the trained modal model sent by the final client, the server can evaluate each trained modal model using evaluation indicators such as accuracy, precision, recall, mean square error, root mean square error, and mean absolute error. If any evaluation indicator of the trained modal model is less than the evaluation indicator threshold, the final client corresponding to the trained modal model is marked as a poor-performing client. Then, the poor-performing client can be improved by allowing the poor-performing client to obtain the feature fusion weights, related feature extraction parameters, and parameters of the inverse diffusion model of other clients.

[0131] It should be noted that the multimodal model obtained using the method of the present application can be applied to actual tasks. For example, if it is necessary to classify organisms using data from multiple modalities, the modal data in the target client are all relevant data of the organisms that need to be classified, such as pictures of the organisms, calls of the organisms, relevant text describing the organisms, etc., and all modal models constructed on the server are models that use data from the corresponding modalities to classify organisms. The multimodal model finally obtained is a model that uses data from multiple modalities to classify organisms. In actual application, the multimodal data of the organisms that need to be classified are input into the multimodal model to obtain the classification results of the organisms, such as whether the organisms are birds, reptiles, arthropods, etc.

[0132] It is worth mentioning that the final fusion features and final time features of the missing client are used to obtain the data of the missing modality, taking into account both the data itself and the time series of the data, thereby improving the accuracy of the data of the missing modality and reducing the situation of missing client data. It avoids the situation where the client with missing modality cannot use the model downloaded from the server and thus cannot perform model training. After the client's modal data is completed, the client can also perform subsequent model training like other clients with complete data, thereby improving the quality of the multimodal model.

[0133] The multimodal federated learning method of this application is illustrated below with reference to a specific example.

[0134] The multimodal model generation system for compensating for modal missingness constructed according to the multimodal federated learning method for compensating for modal missingness of this application is as follows Figure 3 As shown, it includes a central server and multiple clients, such as client 1, client 2, and client N. Among them, client 2 is a missing client and needs to complete its modality. The central server sends the initial modal model to each client. After the client trains the model, it uploads the trained modal model to the central server. The central server aggregates and generates a multimodal model.

[0135] The following is an exemplary description of the multimodal federated learning device provided in this application for compensating for modality loss.

[0136] like Figure 4 As shown, an embodiment of the present application provides a multimodal federated learning device for compensating for modality loss. The multimodal federated learning device 400 for compensating for modality loss includes:

[0137] Acquisition module 401 acquires data of all modalities stored in each target client;

[0138] Determination module 402, based on the data of all modalities stored in each target client, determines missing clients with modality missing data from all target clients;

[0139] Feature extraction module 403 performs feature extraction on the data of each modality of each missing client to obtain multiple data features, fuses all the data features to obtain a final fused feature, extracts the time feature of the data of each modality of the missing client, fuses all the time features to obtain a final time feature, and obtains the data of the missing modality of the missing client based on the final fused feature and the final time feature.

[0140] Training module 404 obtains a modal model corresponding to each modality, treats each target client and each missing client after obtaining data for the missing modality as a final client, and trains the modal model corresponding to the modality using data from all modalities of each final client to obtain a trained modal model for each final client.

[0141] The integration module 405 integrates all trained modal models to obtain a multimodal model.

[0142] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0144] like Figure 5 As shown, an embodiment of the present application provides a terminal device. The terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 5 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above-mentioned method embodiments when executing the computer program D102.

[0145] Specifically, when the processor D100 executes the computer program D102, it obtains data of all modalities stored in each target client, and then determines the missing clients with modal missing data from all target clients based on the data of all modalities stored in each target client, and then performs feature extraction on the data of each modality of the missing client for each missing client to obtain multiple data features, fuses all data features to obtain a final fused feature, and extracts the time feature of the data of each modality, fuses all time features to obtain a final time feature, and obtains the data of the missing modality of the missing client based on the final fused feature and the final time feature, and then obtains the modal model corresponding to the modality for each modality, and regards each target client and each missing client after obtaining the data of the missing modality as the final client, and uses the data of all modalities of each final client to train the modal model corresponding to the modality to obtain the trained modal model of each final client, and finally integrates all trained modal models to obtain a multimodal model. Among them, the final fusion features and final time features of the missing client are used to obtain the data of the missing modality, taking into account both the data itself and the time series of the data, thereby improving the accuracy of the data of the missing modality and reducing the situation of missing client data. It avoids the situation where the client with missing modality cannot use the model downloaded from the server and thus cannot perform model training. After the client's modal data is completed, the client can also perform subsequent model training like other clients with complete data, thereby improving the quality of the multimodal model.

[0146] The processor D100 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0147] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device D10. Furthermore, the memory D101 may include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory D101 may also be used to temporarily store data that has been output or is about to be output.

[0148] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0149] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0150] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a multimodal federated learning method apparatus / terminal device that compensates for modality loss, a recording medium, computer memory, read-only memory (ROM), random access memory (RAM), an electric carrier signal, a telecommunications signal, and a software distribution medium. Examples include a USB flash drive, a removable hard drive, a magnetic disk, or an optical disk. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunications signals.

[0151] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0152] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A multimodal federated learning method for compensating for modality loss, characterized in that: include: Get the data of all modalities stored by each target client; According to the data of all modalities stored in each of the target clients, determining missing clients whose data has modality missing from all the target clients; For each of the missing clients, feature extraction is performed on the data of each modality of the missing client to obtain multiple data features, all the data features are fused to obtain a final fused feature, and time features of the data of each modality of the missing client are extracted, all the time features are fused to obtain a final time feature, and based on the final fused feature and the final time feature, the data of the missing modality of the missing client is obtained; For each of the modalities, a modal model corresponding to the modality is obtained respectively, each of the target clients and each of the missing clients after obtaining the data of the missing modality is used as a final client, and the modal model corresponding to the modality is trained using the data of all modalities of each of the final clients to obtain a trained modal model of each of the final clients; All trained modal models are integrated to obtain a multimodal model.

2. The multimodal federated learning method according to claim 1, characterized in that: The feature extraction is performed on the data of each modality of the missing client to obtain multiple data features, including: By formula: Get the Data characteristics of the data of the modality ; in, , represents the number of modalities of the missing client, Indicates The data features of the first dimension of the modal data, Indicates The data features of the second dimension of the modal data, Indicates The first The data features of the dimensions, express A collection of data features in multiple dimensions.

3. The multimodal federated learning method according to claim 1, characterized in that: The above-mentioned fusion of all data features to obtain the final fusion features includes: Calculate the attention weight between each two modal data, and calculate multiple modal vectors corresponding to each modal data based on all attention weights and all data features; All modal vectors are fused to obtain the final fusion feature.

4. The multimodal federated learning method according to claim 3, characterized in that: The method of calculating multiple modality vectors corresponding to the data of each modality based on all attention weights and all data features includes: By formula: Calculate in The interaction of data from different modalities The modal vector of the data for each mode; in, Indicates the The data of each modality are The attention weight of the data of each modality, Indicates the The data characteristics of the data of each modality, , , The modality number representing the missing client.

5. The multimodal federated learning method according to claim 4, characterized in that: All modal vectors are fused to obtain the final fusion features, including: By formula: Calculate the final fusion features ; in, represents the fused modal vector of the data of the first modality, represents the fused modal vector of the data of the second modality, Indicates The fused modal vector of the data of the modalities, Indicates The fused modal vector of the data of the modalities, Indicates the interaction of the data in the first mode. The modal vector of the data for each mode, Indicated in The interaction of data of the two modalities is described in the following The modal vector of the data for each mode, represents the Hadamard product.

6. The multimodal federated learning method according to claim 1, characterized in that: The using the data of all modalities of each of the final clients to train the modal model corresponding to the modality to obtain the trained modal model corresponding to the modality of each of the final clients includes: For each of the final clients, perform the following steps: Target data of the same modality as the modal model corresponding to the modality is determined from the data of all modalities of the final client, the trained parameters of the modal model are calculated using the target data, and the trained parameters are substituted into the modal model to obtain the trained modal model corresponding to the modality of the final client.

7. The multimodal federated learning method according to claim 6, characterized in that: The calculating the trained parameters of the modal model using the target data includes: By formula: Calculate the The final client corresponds to The parameters of the modal model after training ; in, Indicates the The end client is The target data of the same mode of the modal model, Indicates the The initial parameters of the modal model are , Indicates the number of final clients, , represents the total number of modes, represents the learning rate, represents the number of training rounds, Represents a training operation.

8. The multimodal federated learning method according to claim 7, characterized in that: All trained modal models are integrated to obtain a multimodal model, including: By formula: Calculating multimodal parameters ; in, Indicates the The fusion weights of the modal models, Indicates the The fusion parameters of the modal model.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multimodal federated learning method for compensating for modality loss as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal federated learning method for compensating for modality loss as described in any one of claims 1 to 8 is implemented.

Images