Method and device for realizing heterogeneous missing data processing based on multi-modal federated learning, processor and computer readable storage medium thereof

Abstract

The invention relates to a method for realizing heterogeneous missing data processing based on multi-modal federated learning, which comprises the following steps that: a distributed client receives multi-modal heterogeneous data from a plurality of edge devices, performs defect simulation on the received data, and processes defect data; and the federal learning server receives the model optimization parameters from the plurality of clients, carries out aggregation operation on the received optimization parameters, and downloads and returns the optimization parameters to each client. By adopting the method and device for realizing heterogeneous missing data processing based on multi-modal federated learning, the processor and the computer readable storage medium thereof, the problems of heterogeneity and defects of data acquisition between edge devices can be fully considered, and by improving preprocessing, feature extraction, defect data completion and adversarial learning correction mechanisms, the accuracy of data acquisition is improved, and the accuracy of data acquisition is improved. The efficient fusion and joint modeling of the multi-modal data are realized, so that the training efficiency and the prediction accuracy of the model are improved on the premise of ensuring the data privacy.

Description

Technical Field

[0001] This invention relates to the field of multimodal learning, and more particularly to the field of federated learning. Specifically, it relates to a method, apparatus, processor, and computer-readable storage medium for processing heterogeneous missing data based on multimodal federated learning. Background Technology

[0002] With the advent of the big data era, data acquisition and utilization have become increasingly important. However, data privacy and security issues have also arisen, especially in areas involving personal privacy such as healthcare, finance, and social networks. Protecting data privacy and strengthening data security are urgent problems. Federated learning (FL), as an emerging distributed learning framework, offers an innovative solution to these issues. For example, in the healthcare field, hospitals can share experiences through federated learning without exposing patients' personal information. In the financial industry, banks can improve the accuracy of anti-fraud models through collaborative learning while protecting customer privacy. Federated learning allows multiple participants to jointly train machine learning models while protecting data privacy. By performing computation on local devices, federated learning avoids centralizing sensitive data on a single server, improving both model performance and data security.

[0003] While federated learning offers significant advantages in privacy and data security, most existing FL methods still use single-modal data for training. With the continuous development of edge computing, data generated by different sensors and devices possesses different modalities (e.g., tactile, visual, auditory). Different modalities of data may have different distribution characteristics, dimensions, and types, and this heterogeneity complicates model training. In current technologies, traditional federated learning methods primarily rely on each client completing single-modal model training locally, then uploading the updated parameters to a central server for separate parameter aggregation for each modality. While this approach offers advantages in protecting data privacy, it often struggles to simultaneously address the heterogeneity and defects of data when handling tasks composed of multi-source, heterogeneous modal data. Therefore, research into multi-modal machine federated learning (MMFL) is necessary.

[0004] Currently, multimodal data acquisition typically involves multiple modalities such as images, text, audio, and video. Due to limitations imposed by the sensor characteristics of edge devices, the instability of network transmission, and differences in device hardware performance, the data collected by different clients exhibits significant variations in data format, quality, and modal integrity. This poses a substantial challenge to unified preprocessing, feature extraction, and model training. Current research on MMFL largely focuses on improving model performance and accuracy, with relatively little research addressing data heterogeneity. Existing technologies such as... Figure 2 As shown, the following objective disadvantages exist: (1) Insufficient heterogeneous data processing: When dealing with the data distribution and modal differences between different clients, existing methods mostly adopt a unified preprocessing and feature extraction process, which cannot fully adapt to the diversity of data characteristics of various edge devices; (2) Incomplete defect data completion: The strategy for completing and correcting data defects fails to effectively utilize the contextual information between data, resulting in a large error introduced during the repair process, which affects the subsequent model training effect; (3) Insufficient balance between privacy protection and efficiency: Although federated learning frameworks have a natural advantage in data privacy protection, in the actual parameter aggregation process, due to the differences in computing resources and network conditions of each client, it is easy to cause some clients to update with delays, which in turn affects the real-time performance and accuracy of the global model. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus, processor and computer-readable storage medium for heterogeneous missing data processing based on multimodal federated learning, which has high training efficiency, high prediction accuracy and wide applicability.

[0006] To achieve the above objectives, the present invention provides a method, apparatus, processor, and computer-readable storage medium for processing heterogeneous missing data based on multimodal federated learning, as follows: The main feature of this method for processing heterogeneous missing data based on multimodal federated learning is that the method includes the following steps: (1) The distributed client receives multimodal heterogeneous data from multiple edge devices, simulates defects in the received data, and processes the defect data; (2) The federated learning server receives model optimization parameters from multiple clients, aggregates the received optimization parameters, and then sends them back to each client.

[0007] Preferably, step (1) specifically includes the following steps: (1.1) The client receives multimodal heterogeneous data from multiple edge devices; (1.2) The client simulates defective data caused by data loss in a real-world application scenario, randomly selects a defect rate to generate a defective data mask, and uses the mask to simulate partially lost multimodal data to generate defective data. ; (1.3) Using defective multimodal data, the client uses a modality-specific coding model to extract features from the defective data of each modality and calculate the orthogonal loss function.

[0008] Preferably, step (1) further includes the following steps: (1.4) The client repairs the defective data by using the context information of the feature vector through the defective data completion structure based on the same modality data, and calculates the completion loss function to minimize the difference between the feature vector after completion and the feature vector of the complete data; (1.5) The client corrects the defective data through a modal data correction structure based on adversarial learning mechanism, generates the correction features of the defective data using the comprehensive information of all modalities, and calculates the adversarial loss function by weighted summation with the defective data. (1.6) The client calculates the updated parameters of the model based on the weighted sum of all loss functions and uploads them to the federated learning server.

[0009] Preferably, the multimodal heterogeneous data in step (1.1) includes client identifiers. Data modalities Modal category set Modal Data In this case, missing modal data is set to a zero vector.

[0010] Preferably, in step (1.2), defect data is generated. Specifically: Defect data is generated according to the following formula. : ; in, To simulate a mask for defect data, For modality The data.

[0011] Preferably, the calculation of the orthogonal loss function in step (1.4) is specifically as follows: Calculate the orthogonal loss function using the following formula: ; in, It is the Frobenius norm. For the modalities of the data, It is a set of modal categories.

[0012] Preferably, the calculation of the completion loss function in step (1.4) is specifically as follows: Calculate the completion loss function using the following formula: ; in, For the modalities of the data, For a set of modal categories, For modality Data, Based on Temporal feature coding model, This is the feature data after completion.

[0013] Preferably, the calculation of the adversarial loss function in step (1.5) is specifically as follows: Calculate the adversarial loss function using the following formula: ; in, for function, The regularization coefficient is . For noise distribution, For the true data distribution, To generate a data distribution.

[0014] Preferably, step (1.6) specifically includes the following steps: (1.6.1) Based on the calculated values ​​of the four loss functions, calculate the weighted sum of the loss functions; (1.6.2) Based on the weighted summation of the calculated loss function, calculate the update gradient of the model parameters and optimize the local model parameters through the backpropagation algorithm.

[0015] Preferably, step (2) specifically includes the following steps: (2.1) The federated learning server receives the model parameters uploaded by the client and saves them to the storage according to the client identifier; (2.2) The federated learning server aggregates the model parameters of the clients, extracts the parameter data of K clients from the memory, sums them up and calculates the average value, and integrates the model optimization data of all clients; (2.3) The federated learning server will download the aggregated model parameters to each client in sequence according to the client identifier, and update the client model parameters.

[0016] Preferably, in step (2.2), the client's model parameters are aggregated, specifically as follows: The client's model parameters are aggregated according to the following formula: ; in, These are global parameters for the federated learning server. For the client Model parameters.

[0017] The main feature of this device for processing heterogeneous missing data based on multimodal federated learning is that the device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method described above for processing heterogeneous missing data based on multimodal federated learning.

[0018] The processor for processing heterogeneous missing data based on multimodal federated learning is characterized in that the processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the various steps of the method for processing heterogeneous missing data based on multimodal federated learning.

[0019] The computer-readable storage medium is characterized in that it stores a computer program that can be executed by a processor to implement the various steps of the above-described method for processing heterogeneous missing data based on multimodal federated learning.

[0020] The present invention employs a method, apparatus, processor, and computer-readable storage medium for processing heterogeneous missing data based on multimodal federated learning. This method fully considers the heterogeneity and defects in data collection between edge devices. By improving preprocessing, feature extraction, defect data completion, and adversarial learning correction mechanisms, it achieves efficient fusion and joint modeling of multimodal data, thereby improving the training efficiency and prediction accuracy of the model while ensuring data privacy. Attached Figure Description

[0021] Figure 1 This is a structural diagram of the method for processing heterogeneous missing data based on multimodal federated learning according to the present invention.

[0022] Figure 2 This is a flowchart of the data heterogeneity handling method in existing multimodal federated learning technologies.

[0023] Figure 3 This is a schematic diagram of the defect data completion structure of the client side of the method for processing heterogeneous missing data based on multimodal federated learning according to the present invention.

[0024] Figure 4This is a diagram of the server-side multimodal parameter coordination architecture for the method of heterogeneous missing data processing based on multimodal federated learning according to the present invention.

[0025] Figure 5 This is a flowchart illustrating the training process of the method for processing heterogeneous missing data based on multimodal federated learning according to the present invention. Detailed Implementation

[0026] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.

[0027] The method for processing heterogeneous missing data based on multimodal federated learning of the present invention includes the following steps: (1) The distributed client receives multimodal heterogeneous data from multiple edge devices, simulates defects in the received data, and processes the defect data; (2) The federated learning server receives model optimization parameters from multiple clients, aggregates the received optimization parameters, and then sends them back to each client.

[0028] In a preferred embodiment of the present invention, step (1) specifically includes the following steps: (1.1) The client receives multimodal heterogeneous data from multiple edge devices; (1.2) The client simulates defective data caused by data loss in a real-world application scenario, randomly selects a defect rate to generate a defective data mask, and uses the mask to simulate partially lost multimodal data to generate defective data. ; (1.3) Using defective multimodal data, the client uses a modality-specific coding model to extract features from the defective data of each modality and calculate the orthogonal loss function.

[0029] In a preferred embodiment of the present invention, step (1) further includes the following steps: (1.4) The client repairs the defective data by using the context information of the feature vector through the defective data completion structure based on the same modality data, and calculates the completion loss function to minimize the difference between the feature vector after completion and the feature vector of the complete data; (1.5) The client corrects the defective data through a modal data correction structure based on adversarial learning mechanism, generates the correction features of the defective data using the comprehensive information of all modalities, and calculates the adversarial loss function by weighted summation with the defective data. (1.6) The client calculates the updated parameters of the model based on the weighted sum of all loss functions and uploads them to the federated learning server.

[0030] In a preferred embodiment of the present invention, the multimodal heterogeneous data in step (1.1) includes client identifiers. Data modalities Modal category set Modal Data In this case, missing modal data is set to a zero vector.

[0031] In a preferred embodiment of the present invention, defect data is generated in step (1.2). Specifically: Defect data is generated according to the following formula. : ; in, To simulate a mask for defect data, For modality The data.

[0032] In a preferred embodiment of the present invention, the calculation of the orthogonal loss function in step (1.4) is specifically as follows: Calculate the orthogonal loss function using the following formula: ; in, It is the Frobenius norm. For the modalities of the data, It is a set of modal categories.

[0033] In a preferred embodiment of the present invention, the calculation of the completion loss function in step (1.4) specifically involves: Calculate the completion loss function using the following formula: ; in, For the modalities of the data, For a set of modal categories, For modality Data, Based on Temporal feature coding model, This is the feature data after completion.

[0034] In a preferred embodiment of the present invention, the calculation of the adversarial loss function in step (1.5) is specifically as follows: Calculate the adversarial loss function using the following formula: ; in, for function, The regularization coefficient is . For noise distribution, For the true data distribution, To generate a data distribution.

[0035] In a preferred embodiment of the present invention, step (1.6) specifically includes the following steps: (1.6.1) Based on the calculated values ​​of the four loss functions, calculate the weighted sum of the loss functions; (1.6.2) Based on the weighted summation of the calculated loss function, calculate the update gradient of the model parameters and optimize the local model parameters through the backpropagation algorithm.

[0036] In a preferred embodiment of the present invention, step (2) specifically includes the following steps: (2.1) The federated learning server receives the model parameters uploaded by the client and saves them to the storage according to the client identifier; (2.2) The federated learning server aggregates the model parameters of the clients, extracts the parameter data of K clients from the memory, sums them up and calculates the average value, and integrates the model optimization data of all clients; (2.3) The federated learning server will download the aggregated model parameters to each client in sequence according to the client identifier, and update the client model parameters.

[0037] In a preferred embodiment of the present invention, step (2.2) involves aggregating the model parameters of the client, specifically as follows: The client's model parameters are aggregated according to the following formula: ; in, These are global parameters for the federated learning server. For the client Model parameters.

[0038] The apparatus for processing heterogeneous missing data based on multimodal federated learning of the present invention includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method described above for processing heterogeneous missing data based on multimodal federated learning.

[0039] The processor of the present invention for processing heterogeneous missing data based on multimodal federated learning is configured to execute computer-executable instructions, which, when executed by the processor, implement the various steps of the above-described method for processing heterogeneous missing data based on multimodal federated learning.

[0040] The computer-readable storage medium of the present invention stores a computer program thereon, which can be executed by a processor to implement the various steps of the above-described method for processing heterogeneous missing data based on multimodal federated learning.

[0041] To address the modality missing problem in existing federated learning techniques involving multi-source heterogeneous data, this invention discloses a method for processing heterogeneous missing data based on multimodal federated learning. It employs an architecture where distributed clients and a federated learning server work collaboratively. The distributed clients receive multimodal heterogeneous data from edge devices and utilize LSTM to construct a temporal feature encoding model to extract multimodal features. Simultaneously, a Transformer-based defect data completion strategy and adversarial learning mechanisms are combined to process the defective data. The federated learning server uses a parameter aggregation model, based on FedAVG, to integrate parameters from various clients to update the global model.

[0042] The core of this application lies in a privacy-preserving, distributed model training method specifically designed to address the inherent modality gaps in client-side models within federated learning scenarios. Its technical solution is based on a federated learning architecture, where distributed clients and a federated learning server collaborate. After receiving local multimodal data, each client actively simulates data defects, then uses LSTM for temporal feature extraction, and subsequently uses a Transformer-based same-modality data completion module and an adversarial learning-based multimodal correction module to repair and enhance the missing data. Finally, the federated learning server aggregates the model parameters from each client using a federated averaging algorithm (FedAVG) to update the global model, while the original data remains locally on each client.

[0043] This invention includes the following modules: Module 1, Distributed Client, is used to receive multimodal heterogeneous data from multiple edge devices, simulate defects in the received data, and process the defect data. Its structure is as follows: Figure 3 As shown, this is to fill in the missing parts of the multimodal data, and at the same time, the model update parameters are uploaded to the federated learning server during the training process. The distributed client constructs a temporal feature encoding model using a Long Short Term Memory Network (LSTM) to extract multimodal features, and uses a Transformer-based defect data completion strategy and adversarial learning mechanism to process defect data.

[0044] Module 2, the Federated Learning Server, is used to receive model optimization parameters from multiple clients, aggregate the received optimization parameters, and then return them to each client, such as... Figure 4 As shown, this is used to update the model parameters.

[0045] The federated learning server employs a parameter aggregation model, integrating client parameters based on the Federated Averaging (FedAVG) algorithm to update the global model. This invention trains and optimizes the model using simulated defective data, combining feature complementarity loss and adversarial learning loss functions to effectively improve data integrity and model robustness. It significantly enhances the performance and efficiency of multimodal federated learning while protecting user privacy.

[0046] In Module 1, the distributed client processes the received multimodal heterogeneous data, such as... Figure 5 As shown, it is configured to include the following steps: S1. The client receives multimodal heterogeneous data from multiple edge devices. If the client... The received data lacks modalities. If the data is given, then the data for that modality will be used. filling; S2. The client simulates defective data caused by data loss in a real-world application scenario, randomly selects a defect rate to generate a defective data mask, and uses the mask to simulate some of the lost multimodal data. S3. Based on the defective multimodal data obtained in S2, the client uses a modality-specific coding model to extract features from the defective data in each modality and calculate the orthogonal loss function. The advantage of this invention is that, in S1, the multimodal heterogeneous data received by the distributed client includes: client identifier. Data modalities Modal category set Modality Data ,satisfy In this case, missing modal data is set to a zero vector.

[0047] The advantage of this invention is that, in S2, the mask of the simulated defect data... According to the preset defect rate Randomly generated, and defect data generated. The following formula can be used for calculation: ; The advantage of this invention is that, in S3, the multimodal data feature extraction model is configured as follows: based on Temporal feature coding model Different features extracted based on different modalities When training and optimizing parameters, the loss function of the encoding model is based on the complementary nature of features. The orthogonal loss function is calculated as follows: ; in, The L2 norm is used to optimize the ability of the defect data completion model to repair defective data by minimizing the difference between the feature vector after completion and the feature vector of the complete data.

[0048] In Module 1, the defect data completion structure is configured as follows: S4. The client uses the context information of the feature vector to repair the defective data by completing the defective data based on the same modality data, and calculates the completion loss function to minimize the difference between the feature vector after completion and the feature vector of the complete data. S5. The client corrects the defective data through a modal data correction structure based on adversarial learning mechanism. It generates corrected features of the defective data by using the comprehensive information of all modalities, then sums them with the defective data in a weighted manner and calculates the adversarial loss function.

[0049] S6. The client calculates the updated parameters of the model based on the weighted sum of all loss functions and uploads them to the federated learning server.

[0050] The advantage of this invention is that, in S4, the defect data completion structure based on the same modal data is configured as follows: based on Defect data completion model Based on the modal feature data, calculate the completed feature data. When training and optimizing parameters, the method uses a loss function that optimizes the model's repair capability based on the difference between the completed and full data. The calculation formula is as follows: ; The advantage of this invention is that, in S5, the modality data correction structure based on the adversarial learning mechanism is configured as follows: based on Data defect rate calculation model with Multilayer Perceptron (MLP) Based on feature data Calculate the data defect rate The predicted data defect rate is derived based on the data defect rate calculation model. When training and optimizing parameters, the method is based on the defect rate of the predicted data. Defect rate compared to real data loss function ;based on Auxiliary feature generator Based on auxiliary feature generator Generated modal supplement features Modal supplementation features The completion feature is calculated by weighted summation. Its calculation formula is ; When training and optimizing parameters, the method is based on a feature discriminator. To distinguish the differences between the completed features and the original features, the adversarial loss function Includes a gradient penalty term, and calculates the loss function. The calculation formula is as follows: ; in, for function, The regularization coefficient is . For noise distribution, For the true data distribution, To generate a data distribution.

[0051] In S6, the model parameter update operation includes the following steps: S7. Based on the values ​​of the four loss functions calculated in S3, S4, and S5, the client calculates the weighted sum of the loss functions. S8. Based on the sum of the loss functions calculated in S7, the client calculates the update gradient of the model parameters and optimizes the local model parameters through the backpropagation algorithm.

[0052] In Module 2, the federated learning server includes the following steps: S9. The federated learning server receives the model parameters uploaded by the client and saves them to the storage according to the client identifier; S10. The federated learning server aggregates the model parameters of the clients, extracts the parameter data of K clients from the memory, and calculates the average value after summing them to integrate the model optimization data of all clients. S11. The federated learning server will sequentially download and return the aggregated model parameters to each client according to the client identifier, and update the client model parameters.

[0053] The advantage of this invention is that, in S10, the model parameter aggregation operation can be calculated using the following formula: ; in, These are global parameters for the federated learning server. For the client Model parameters.

[0054] This invention simulates data loss in real-world application scenarios to obtain incomplete multimodal data. Then, by combining feature complementation loss functions and adversarial learning loss functions, the data is completed and corrected, further enhancing the model's robustness to data defects and enabling it to maintain good performance even with incomplete data. Therefore, it overcomes the shortcomings of existing technologies, effectively improving data integrity and consistency while protecting user privacy, and significantly enhancing the model's generalization ability and prediction accuracy.

[0055] In a specific embodiment of the present invention, the following steps are included: S1, the client receives multimodal heterogeneous data from multiple edge devices. The data content includes: client identifier. Data modalities Modal category set Modality Data It satisfies the following formula: ; in, Representing modes Data dimensions The number of modalities received by each client is typically determined by the federated learning server, and may vary. If the client... The received data lacks modalities. The data, then the data of this modality Will be placed as The dimension is the same as the data dimension of other modalities, that is .

[0056] S2, to better reflect the data loss and defective data situations in real-world applications, introduces the concept of defect rate during model training. (Each client...) A defect rate will be generated. This is used to simulate the missing rate of defect data. Defect rate The range of values ​​is This represents the missing rate of defective data; a higher value indicates a higher missing rate, meaning more severe data loss. Based on the defect rate... Based on the defect rate, when training model parameters, each client... A defect data mask will be generated. This is used to simulate missing defect data. Defect data mask. Each element The range of values ​​is It can be calculated using the following formula: ; in, ( () is an independent random variable that takes the value 0 or 1, and the probability of taking the value 0 is... The probability of taking the value 1 is , can be represented as: ; ; During training with complete data, the distributed client performs a masking operation on the received multimodal heterogeneous data to obtain defective multimodal data. It can be expressed as follows: ; In this way, the model can be exposed to simulated defective data during training, thereby enhancing the model's robustness to data loss and enabling it to better handle and learn when faced with data loss problems in real-world applications, thus improving the model's generalization ability and reliability.

[0057] S3, based on the multimodal data acquired in S2, is used by the client through a modality-specific encoding model. For defective multimodal data Feature extraction is performed, and the extracted features .in, For modality Specific coding models can delve deeper into the valuable information contained within data of different modalities. For example, for video data, a coding model can extract features such as visual texture, shape, and color; for audio data, it can extract features such as frequency, amplitude, and rhythm. This coding model is based on... Design, modality Temporal feature encoding of data can better capture the patterns and correlations of data changes over time, thereby further improving the effectiveness of feature extraction. To achieve feature vectors extracted from different modalities using a modality-specific encoding model that satisfy the property of information complementarity, the client will use a loss function that evaluates the orthogonality between different modalities. It can be calculated using the following formula: ; in, The Frobenius norm is used. Through the design of orthogonal loss functions, the multimodal feature vectors extracted by the encoder can complement each other, providing richer information support for subsequent analysis tasks.

[0058] In another instance, in S2, for real-world application scenarios, the data received by the client is assumed to have certain defects, and the client does not need to perform the step of simulating data defects when processing the data.

[0059] In another example, in S3, a modality-specific coding model for the client is used, tailored to the specific application scenario. The extracted multimodal data is the raw data received directly from the edge device by the client. This is not data that has undergone simulation and defect processing. Furthermore, the encoding model does not require calculating an orthogonal loss function when processing data. The model directly performs feature extraction operations on the original data to obtain... This processing method can more realistically reflect the original characteristics of the data collected by edge devices, avoiding additional errors that may be introduced by repeated simulation of data defects, thereby improving the accuracy and reliability of data processing.

[0060] In Module 1, the client's defect data completion structure is configured to include the following steps: S4, the client completes the defect data using a defect data completion structure based on the same modality data, and the defect data is then processed by the defect data completion model. By utilizing the contextual information of feature vectors, defective data is repaired, and the completed features... .in, Based on A defect data completion model. During model training, to optimize the defect data completion model's ability to repair defective data, the client will use the loss function of the defect data completion model. It can be calculated using the following formula: ; in, The L2 norm is used to optimize the ability of the defect data completion model to repair defective data by minimizing the difference between the feature vector of the completed data and the feature vector of the complete data. S5, the client corrects defective data through a modal data correction structure based on adversarial learning, and the defective data is processed by a data defect rate calculation model. Calculate the modes Defect rate of defect data To optimize the computational power of the data defect rate calculation model during training, a loss function for the model was designed. It can be calculated using the following formula: ; in, Representing the L2 norm, this method optimizes the defect rate calculation model's ability to calculate defect rates by minimizing the difference between the defect rate in the defect data and the true defect rate. (Defect data correction model) By utilizing the comprehensive information from all modalities, corrective features for defect data are generated. Then, with the completion features of the defect data. We perform weighted summation to obtain the final multimodal data features. The calculation formula is as follows: ; During model parameter training, in order to optimize the defect data correction model's ability to correct defect data, a feature discriminator-based approach is used. To distinguish the differences between the completed features and the original features, the client will use the loss function of the defective data correction model. This approach aims to optimize model performance through adversarial learning. The loss function consists of three parts: a discriminant loss function, a gradient penalty term for the discriminator, and a gradient penalty term for the generator. It aims to improve the model's ability to identify and correct defective data by minimizing the difference between the generated and real data distributions. By calculating the gradient norms of the discriminator and generator outputs, these penalty terms effectively suppress gradient explosion or vanishing problems that may occur during model training, further enhancing the model's stability and robustness. This process can be calculated using the following formula: ; in, for function, The regularization coefficient is . For noise distribution, For the true data distribution, To generate a data distribution; S6, per client The local model parameters will be updated, and the updated parameters will be uploaded to the federated learning server. This process includes the following steps: S7, based on the values ​​of the four loss functions calculated in S3, S4, and S5, the client calculates the weighted sum of the loss functions. It can be expressed as follows: ; in, These are the weight coefficients for the four loss functions. These coefficients are typically adjusted manually based on the specific scenario of each client and the environment of the edge device. By setting these weight coefficients appropriately, it can be ensured that the model achieves optimal performance in different environments. S8, the total loss calculated based on S7. Client The gradient of the model parameters can be calculated. Then, the model update parameters are calculated using the backpropagation algorithm. It can be expressed as follows: ; in, For learning rate, These are the local model parameters. Learning rate. The choice of parameters has a significant impact on the convergence speed and stability of the model, and usually needs to be adjusted according to the specific problem and dataset. In this way, each client can update the model parameters locally, thus achieving distributed training of the model. After obtaining the updated model parameters, the client will upload the parameters to the federated learning server.

[0061] In another example, in S4, for a real-world application scenario, the client lacks complete multimodal data as a repair reference; therefore, the client does not need to calculate the loss function. To train the model's data repair capabilities.

[0062] In another example, in S5, for a real-world application scenario, the client cannot obtain the defect rate of the received data, therefore the client does not need to calculate the loss function. Similarly No calculation is required.

[0063] In Module 2, the federated learning server is configured to include the following steps: S9, the federated learning server receives model parameters uploaded by the client and saves them to the storage according to the client identifier; In S10, the federated learning server aggregates the model parameters from the clients, extracts the parameter data from K clients from memory, sums them, and calculates the average to obtain the global model parameters. Based on the comprehensive model optimization data from all clients, it can be expressed as follows: ; S11, the federated learning server will transfer global model parameters The data is sequentially downloaded and returned to each client according to the client identifier, updating the local model parameters. It can be expressed as follows: ; The embodiments of the present invention are as follows: Step one: The distributed client receives multimodal heterogeneous data from multiple edge devices. This data includes, but is not limited to, images, text, audio, and video. Each data piece is accompanied by a client identifier to clearly identify the data source; simultaneously, the data's modality type, modality category set, and specific data content are all recorded in detail. After receiving the data, the distributed client first performs preliminary preprocessing, including data format standardization and data normalization, to ensure smooth subsequent processing.

[0064] Step two: The distributed client extracts features from the defective multimodal data using a modality-specific encoding model, obtaining feature vectors for each modality. Multiple encoders with the same design architecture are used for different modalities, with each encoder extracting features only for a single modality. These encoders are all based on... Although their architectures are optimized for different modal data, their underlying structure remains consistent to ensure that the powerful sequence information capture capabilities of LSTM can be fully utilized when processing different modal data.

[0065] Step three: The distributed client repairs defective data using a defect data completion structure based on same-modal data. This structure utilizes the contextual information of the feature vectors to complete the defective data. In this embodiment, a corresponding defect data completion model is designed based on the characteristics of multimodal data, which can effectively repair defective data of different modalities, resulting in more complete and accurate feature vectors after completion.

[0066] Step four: The distributed client corrects the defective data using a modal data correction structure based on adversarial learning. First, the defect rate of the modal defective data is calculated using a data defect rate calculation model. Then, corrected features of the defective data are generated using comprehensive information from all modalities. Finally, the corrected features and the completed features of the defective data are weighted and summed to obtain the final multimodal data features. The weights of the summation are dynamically adjusted based on the calculated defect rate of the modal defective data to ensure the accuracy and robustness of the final features.

[0067] Specifically, this invention utilizes a distributed client to preprocess multimodal data collected from various edge devices, employs an improved coding model to accurately extract features from defective data, and then combines a defective data completion and correction module to achieve automatic repair of incomplete data and high-quality feature fusion. The resulting global model shows significant improvements in both generalization ability and prediction accuracy.

[0068] This invention enables effective learning under privacy protection: under the premise that data cannot legally leave the local area, collaborative modeling across data sources is achieved through a federated architecture.

[0069] This invention achieves inherent robustness to missing data: through “simulated defect” training and “adversarial correction” mechanisms, the model is optimized for data incompleteness from the initial design stage, making it more stable and reliable in real-world scenarios with severe data missing, and fundamentally improving the model’s generalization ability in truly imperfect environments.

[0070] This invention solves technical problems from different dimensions, effectively addressing the two common and intertwined problems of "data silos" and "data gaps" in real-world applications.

[0071] This invention has at least the following beneficial effects:

[0072] Firstly, technological innovation: This embodiment receives heterogeneous multimodal data through a distributed client and uses a modality-specific encoding model for feature extraction, fully utilizing the characteristics of each modality to extract high-quality feature vectors. Simultaneously, a defective data completion module based on same-modality data and a modality data correction module based on adversarial learning mechanisms repair and correct defective data, effectively improving data integrity and accuracy. Furthermore, the model incorporates an adaptive loss weight adjustment mechanism based on data defect status and dynamic modality missing rate, enabling flexible adjustment of training strategies according to different data scenarios, significantly improving training stability and convergence speed.

[0073] Secondly, its practical innovation lies in its efficient aggregation of optimization parameters uploaded from various clients via a federated learning server. Employing a dynamic weight adjustment strategy, it ensures both the real-time and accurate updating of model parameters while respecting data privacy and the limitations of edge computing resources. Furthermore, relying on a multi-layered, phased data fusion strategy, this model can deeply explore the potential correlations between various modalities, achieving highly integrated feature representations and providing more comprehensive and accurate information for subsequent decision-making. In addition, by directly processing raw data on distributed clients and employing a federated parameter aggregation method, this model effectively avoids centralized data transmission, significantly reducing the risk of privacy leaks, while also complying with relevant national and international data protection standards.

[0074] Thirdly, application innovation: This invention can be widely applied in fields such as intelligent security, healthcare, intelligent transportation, and industrial automation. For example, in intelligent security, by integrating video surveillance and facial recognition technologies to process and analyze multimodal data, efficient security monitoring and anomaly detection can be achieved. In the healthcare field, by combining patients' medical imaging data and medical record text data, the accuracy and efficiency of disease diagnosis can be improved. Thanks to technological innovation and greater practicality, this model has broad development prospects in multiple fields.

[0075] For the specific implementation scheme of this embodiment, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0076] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0077] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0078] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0079] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0080] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The corresponding program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0081] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0082] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0083] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0084] The present invention employs a method, apparatus, processor, and computer-readable storage medium for processing heterogeneous missing data based on multimodal federated learning. This method fully considers the heterogeneity and defects in data collection between edge devices. By improving preprocessing, feature extraction, defect data completion, and adversarial learning correction mechanisms, it achieves efficient fusion and joint modeling of multimodal data, thereby improving the training efficiency and prediction accuracy of the model while ensuring data privacy.

[0085] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method for implementing heterogeneous missing data processing based on multi-modal federated learning, characterized in that, The method includes the following steps: (1) The distributed client receives multimodal heterogeneous data from multiple edge devices, simulates defects in the received data, and processes the defect data; (2) The federated learning server receives model optimization parameters from multiple clients, aggregates the received optimization parameters, and then sends them back to each client.

2. The method for processing heterogeneous missing data based on multi-modal federated learning according to claim 1, wherein, Step (1) specifically includes the following steps: (1.1) The client receives multimodal heterogeneous data from multiple edge devices; (1.2) The client simulates defective data caused by data loss in the actual application scenario, generates a defective data mask with a random defect rate, and simulates the partially lost multi-modal data using the mask to generate defective data ; (1.3) Using defective multimodal data, the client uses a modality-specific coding model to extract features from the defective data of each modality and calculate the orthogonal loss function.

3. The method for processing heterogeneous missing data based on multi-modal federated learning according to claim 2, characterized in that, Step (1) further includes the following steps: (1.4) The client repairs the defective data by using the context information of the feature vector through the defective data completion structure based on the same modality data, and calculates the completion loss function to minimize the difference between the feature vector after completion and the feature vector of the complete data; (1.5) The client corrects the defective data through a modal data correction structure based on adversarial learning mechanism, generates the correction features of the defective data using the comprehensive information of all modalities, and calculates the adversarial loss function by weighted summation with the defective data. (1.6) The client calculates the updated parameters of the model based on the weighted sum of all loss functions and uploads them to the federated learning server.

4. The method for processing heterogeneous missing data based on multimodal federated learning according to claim 2, characterized in that, The multi-modal heterogeneous data of step (1.1) comprises a client identification , a modal category of data , a modal category set , a modal of data , wherein the missing modal data is set to a zero vector.

5. The method for processing heterogeneous missing data based on multimodal federated learning according to claim 2, characterized in that, In step (1.2), defect data is generated. Specifically: Defect data is generated according to the following formula. : ； in, To simulate a mask for defect data, For modality The data.

6. The method for processing heterogeneous missing data based on multimodal federated learning according to claim 2, characterized in that, The calculation of the orthogonal loss function in step (1.4) is as follows: Calculate the orthogonal loss function using the following formula: ； in, It is the Frobenius norm. For the modalities of the data, It is a set of modal categories.

7. The method for processing heterogeneous missing data based on multimodal federated learning according to claim 3, characterized in that, The calculation of the completion loss function in step (1.4) is as follows: Calculate the completion loss function using the following formula: ； in, For the modalities of the data, For a set of modal categories, For modality Data, Based on Temporal feature coding model, This is the feature data after completion.

8. The method for processing heterogeneous missing data based on multimodal federated learning according to claim 3, characterized in that, The calculation of the adversarial loss function in step (1.5) is as follows: Calculate the adversarial loss function using the following formula: ； in, for function, The regularization coefficient is . For noise distribution, For the true data distribution, To generate a data distribution.

9. The method for processing heterogeneous missing data based on multimodal federated learning according to claim 3, characterized in that, Step (1.6) specifically includes the following steps: (1.6.1) Based on the calculated values ​​of the four loss functions, calculate the weighted sum of the loss functions; (1.6.2) Based on the weighted summation of the calculated loss function, calculate the update gradient of the model parameters and optimize the local model parameters through the backpropagation algorithm.

10. The method for processing heterogeneous missing data based on multimodal federated learning according to claim 1, characterized in that, Step (2) specifically includes the following steps: (2.1) The federated learning server receives the model parameters uploaded by the client and saves them to the storage according to the client identifier; (2.2) The federated learning server aggregates the model parameters of the clients, extracts the parameter data of K clients from the memory, sums them up and calculates the average value, and integrates the model optimization data of all clients; (2.3) The federated learning server will download the aggregated model parameters to each client in sequence according to the client identifier, and update the client model parameters.

11. The method for processing heterogeneous missing data based on multimodal federated learning according to claim 10, characterized in that, In step (2.2), the client's model parameters are aggregated, specifically as follows: The client's model parameters are aggregated according to the following formula: ； in, These are global parameters for the federated learning server. For the client Model parameters.

12. An apparatus for processing heterogeneous missing data based on multimodal federated learning, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for heterogeneous missing data processing based on multimodal federated learning as described in any one of claims 1 to 11.

13. A processor for processing heterogeneous missing data based on multimodal federated learning, characterized in that, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for heterogeneous missing data processing based on multimodal federated learning as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method for processing heterogeneous missing data based on multimodal federated learning as described in any one of claims 1 to 11.

Images