A multi-modal health data fusion and privacy protection system and method based on federated learning

Abstract

The application discloses a kind of multi-modal health data fusion and privacy protection system and method based on federal learning, belong to health medical big data and privacy computing technical field.The system includes: a plurality of local data nodes deployed in user terminal, for processing multi-modal health data such as tongue image, physiological signal locally;Federal learning coordination server, for coordinating the training and safe aggregation of global model;Privacy-enhanced computing module integrated in the above node, model parameters are encrypted and noisy by homomorphic encryption and differential privacy technology;And cross-node joint analysis application module and block chain storage and audit module.The method makes original data participate in joint modeling without leaving local through federal learning framework, fundamentally protects user privacy;Through homomorphic encryption, ciphertext aggregation is realized, and differential privacy is combined to provide mathematically provable privacy guarantee;Through block chain, process is auditable.The application solves the contradiction between health data "value fusion" and "privacy protection", and provides a technical basis for safe, compliant health big data analysis.

Description

Technical Field

[0001] This invention relates to the fields of health and medical big data, artificial intelligence and privacy computing technology, specifically to a system and method based on federated learning framework for securely fusing, jointly modeling and mining value from multimodal health data distributed across different user terminals, while achieving strict privacy protection. Background Technology

[0002] With the widespread adoption of wearable devices and mobile healthcare apps, personal health data is experiencing explosive growth, encompassing multimodal information such as physiological indicators (heart rate, blood pressure), behavioral data (exercise, sleep), imaging data (tongue and facial images), and genetic data. This data is of immense value for disease risk prediction, personalized health interventions, and medical research. However, current health data applications face two core contradictions: The contradiction between "data silos" and "value integration": Health data is highly dispersed across different entities such as personal mobile phones, smart devices, and medical institutions, forming data silos. The limited data volume and incomplete dimensions of a single entity make it difficult to train accurate AI models. Centralized aggregation, however, presents another contradiction.

[0003] The contradiction between "data aggregation" and "privacy leakage": Health data is highly sensitive personal information and is subject to strict regulation by laws such as the Personal Information Protection Law and the Data Security Law. The traditional model of centralizing raw data for processing on a central server poses significant risks of leakage, misuse, and compliance, resulting in low user willingness to authorize and high compliance costs for enterprises.

[0004] While some existing technologies mention data anonymization and encrypted storage, they do not fundamentally solve the problem that raw data must leave the domain before federated computation can be performed. Federated learning, as an emerging privacy-preserving computing technology, offers a possibility of resolving this contradiction with its "data remains stationary, model moves" concept. However, current federated learning applications are mostly concentrated on single-type data (such as word frequency in mobile input methods), lacking systematic integration solutions for complex, heterogeneous, and multimodal health data, and also have shortcomings in incentive mechanisms, compliance auditing, and cross-scenario applications.

[0005] Therefore, there is an urgent need for a complete technical solution that can systematically address the secure fusion, joint analysis, and value realization of multimodal health data under the premise of privacy protection. Summary of the Invention

[0006] This invention aims to address the technical problem in existing technologies where multimodal health data is difficult to effectively integrate, jointly model, and extract value from while ensuring user privacy and data security. Specifically, it includes: How to perform feature alignment and fusion of distributed multimodal health data without aggregating the original data; How to design a federated learning architecture suitable for health data, supporting heterogeneous device participation and efficient model aggregation; How can we combine homomorphic encryption, differential privacy, and other technologies to provide multi-level privacy and security enhancements during the federated learning process? How to establish a credible incentive mechanism and audit traceability system to ensure the rights and compliance of system participants.

[0007] To achieve the above objectives, the present invention provides a multimodal health data fusion and privacy protection system based on federated learning, characterized in that it includes: Multimodal local data nodes: Deployed on user terminal devices, used to collect, store and process users' local multimodal health data. The multimodal health data includes at least: tongue / facial image data collected by smart terminal cameras, physiological time-series data collected by wearable devices, lifestyle data actively reported by users, and medical record data synchronized from authorized institutions; each local node is equipped with a local feature extraction module and a local model training unit. Federated Learning Coordination Server: Communicates with each local data node to coordinate the federated learning process; it includes: Global Model Management Unit: Responsible for initializing the global machine learning model and distributing model parameters to each local node; Secure aggregation unit: Receives encrypted local model parameter updates (such as gradients and weights) uploaded by each local node, and uses an encrypted aggregation algorithm to fuse them to generate an updated global model; Incentive mechanism management unit: Calculates and allocates incentive points based on the contribution of each local node to the global model (such as data quality, training time, and model improvement); Privacy-enhancing computing module: integrated into the federated learning coordination server and / or local data node, used to apply privacy protection at each stage of federated learning, and includes at least: Homomorphic encryption unit: Homomorphically encrypts the model parameter updates uploaded by the local node, enabling the secure aggregation unit to complete the aggregation calculation in ciphertext state; Differential privacy addition unit: Add random noise that conforms to the definition of differential privacy to the model parameters before local model training or parameter upload; Cross-node joint analysis application module: Based on the global model or intermediate features obtained by federated learning training, it provides privacy-preserving joint analysis services, including population health trend analysis, disease risk prediction and personalized intervention plan generation; Blockchain Evidence Storage and Auditing Module: Used to store key events in the federated learning process (such as global model version, aggregation operation hash, incentive allocation record) on the blockchain, providing tamper-proof audit traceability.

[0008] Compared with the prior art, the present invention has the following beneficial effects: Fundamental privacy protection: Adopting a federated learning framework, users' raw multimodal health data is always stored locally on their devices without being uploaded to any central server, eliminating the risk of data leakage at the source and perfectly complying with the regulatory requirement of "data not leaving the domain".

[0009] Efficient multimodal fusion: By designing a local feature extraction module, heterogeneous multimodal data (images, time-series signals, structured records) are transformed into unified feature vectors locally. Then, federated learning is used to fuse these vectors at the parameter level, solving the challenge of joint modeling of cross-modal data. Experimental results show that the gated attention fusion method of this invention improves the AUC value of the model by approximately 18% compared to the single tongue image data model and approximately 22% compared to the single heart rate data model in the hypertension risk prediction task.

[0010] Multi-layered security reinforcement: Innovatively combining federated learning with homomorphic encryption and differential privacy. Homomorphic encryption ensures the security of parameter transmission and aggregation processes; differential privacy provides strict mathematical privacy guarantees, making it impossible to deduce individual information even if the server is attacked, forming a defense-in-depth system. With a differential privacy parameter of ε=0.5, the probability of an attacker successfully inferring a specific health state of an individual is less than 60%, while the global model prediction accuracy loss does not exceed 2%.

[0011] Sustainable ecosystem building: Through a contribution-based intelligent incentive mechanism, users are encouraged to contribute high-quality data to participate in federated training, and the value of the data is returned to users in the form of points, rights and benefits, forming a positive cycle of "contribution-benefit".

[0012] Trustworthiness and Compliance: By using blockchain technology to record key operations throughout the federated learning process, the process is made transparent, auditable, and tamper-proof, providing a technological foundation for meeting the stringent compliance audit requirements in the healthcare field. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention.

[0014] Figure 2 Workflow diagram for enhanced computing for federated learning and privacy.

[0015] Figure 3 This is a schematic diagram illustrating the composition of a multimodal local data node.

[0016] Figure 4This is a flowchart for incentive mechanism management.

[0017] Figure 5 This is a schematic diagram of a cross-node joint analysis application.

[0018] Figure 6 A data flow diagram for blockchain evidence storage and auditing. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0020] like Figure 1 As shown, this system adopts a "centralized coordination, edge computing" architecture. Thousands of multimodal local data nodes (such as users' smartphones and home health terminals) serve as edge nodes, holding the raw data. One or more federated learning coordination servers act as the central coordinator. Furthermore, the system includes cross-node collaborative analysis application modules for end users or institutions, as well as a blockchain-based evidence storage and auditing module (which can be built on a consortium blockchain) providing underlying trust support.

[0021] Taking a user's smartphone as an example, local nodes include: Data acquisition layer: Uses the phone's camera to capture tongue images under standard light source; connects to a smart bracelet via Bluetooth to obtain time-series data such as heart rate and sleep stages; guides users to input logs of their diet and emotions through the APP interface.

[0022] Local storage and preprocessing layer: After data is de-identified (name and ID number removed), it is stored using a local key for encryption. Preprocessing includes ROI cropping and standardization of tongue images, filtering of physiological data, and feature segmentation extraction.

[0023] Local Feature Extraction Module: For tongue images, a lightweight convolutional neural network (CNN) is run, outputting a 256-dimensional tongue image feature vector. For one week of heart rate variability (HRV) data, time-domain and frequency-domain features are extracted to form a feature vector. The feature vectors from multiple modalities are concatenated to form a local multimodal fusion feature vector.

[0024] Local model training unit: Receives a global model (such as a hypertension risk prediction neural network) from the coordination server. It is trained using local multimodal fusion feature vectors and labels (if any), and the parameter updates (gradients) of the local model are computed.

[0025] Privacy-enhancing client: Before uploading parameter updates, the differential privacy-adding unit is invoked to add Laplacian noise to the gradient. Then, the homomorphic encryption unit is invoked to encrypt the noisy gradient using the coordinating server's public key.

[0026] The workflow of the coordination server is as follows: Task release and model initialization: The server releases a training task (such as "train a fatigue prediction model based on tongue image and heart rate data"), initializes a model architecture, and distributes it to the participating nodes.

[0027] Secure aggregation: The server receives encrypted gradients from multiple local nodes. The secure aggregation unit utilizes the additive homomorphism of homomorphic encryption to directly perform a weighted average of the gradients on the ciphertext. After aggregation, the gradients are decrypted using a private key and used to update the global model.

[0028] Incentive Mechanism Operation: The incentive mechanism management unit calculates the contribution based on the quality of gradients uploaded by each node during the current training cycle (such as gradient magnitude and similarity to the aggregated gradient) and the amount of data, and updates the incentive points account of each node. Points can be used to redeem health reports, consultation services, or compliant digital benefits.

[0029] Model iteration: The updated global model is redeployed to start a new round of training until the model converges or reaches the predetermined number of rounds.

[0030] The trained global model itself is an asset that incorporates collective knowledge. Applications include: Group trend insight: The server can distribute a global "feature extractor" model to each node. Each node uses this model to process local data and uploads encrypted feature vector statistics (such as mean and variance). The server then aggregates these data to analyze group health trends without needing to obtain individual features.

[0031] Personalized services: Users can input the latest local data into the global model to obtain personalized risk predictions (e.g., "Your risk of fatigue is high in the coming week") and receive corresponding intervention suggestions (e.g., "It is recommended to increase deep sleep duration to X hours").

[0032] Each global model version update and each distribution of incentive points generates a transaction for its key parameters (such as model hash and point transaction ID), which is then signed by the coordinating server and submitted to the blockchain network. Any participant can query the on-chain records to verify the authenticity and fairness of the federated learning process, meeting compliance audit requirements.

[0033] Through the above system and method, this invention activates the value of massive, scattered health data that has been dormant, while ensuring absolute security of user privacy, and provides innovative technical infrastructure for fields such as precision medicine, health management, and insurtech.

Claims

1. A multimodal health data fusion and privacy protection system based on federated learning, characterized in that, include: Multiple multimodal local data nodes are deployed on user terminals to store and process users' raw multimodal health data locally and to perform local model training. The federated learning coordination server communicates with each of the local data nodes and is used to distribute the global model, securely aggregate the local model parameter updates uploaded by each node to update the global model, and manage the federated learning process. The privacy-enhancing computation module is used to encrypt and / or add noise to model parameter updates during federated learning to achieve privacy protection. The cross-node joint analysis application module provides health data analysis services with privacy protection based on the global model or aggregated features obtained by federated learning training. The blockchain evidence storage and auditing module is used to record key operations during the federated learning process on the blockchain for evidence storage.

2. The local data node according to claim 1 includes a local feature extraction module, used to convert the original multimodal health data into a unified multimodal fusion feature vector through a gated attention mechanism, wherein, The multimodal fusion feature vector is calculated using the formula F_fused = Σ(α_i × F_i), where α_i is the attention weight of the i-th modality and F_i is the feature vector of the i-th modality.

3. The system according to claim 1, characterized in that, The federated learning coordination server includes: The global model management unit is used for initializing and versioning global machine learning models; The secure aggregation unit uses a homomorphic encryption algorithm to perform aggregate calculations in ciphertext state on the received encrypted local model parameter updates; The incentive mechanism management unit calculates and allocates incentive points based on the contribution of each local data node in federated learning.

4. The system according to claim 3, characterized in that, The privacy-enhancing computing module includes a homomorphic encryption unit deployed on a local data node, used to encrypt local model parameter updates using the public key of the federated learning coordination server before uploading; the secure aggregation unit uses the homomorphic property of homomorphic encryption to complete ciphertext aggregation, and then decrypts it with a private key to obtain the aggregation result.

5. The system according to claim 1 or 3, characterized in that, The privacy-enhancing computation module includes a differential privacy addition unit, which is deployed on a local data node. It is used to add random noise that conforms to the differential privacy definition to the model parameter update after the local model training is completed and before the parameters are uploaded, wherein the privacy budget ε is set to 0.1 to 0.

5.

6. The system according to claim 3, characterized in that, The incentive mechanism management unit calculates the contribution based on: the data quality score of the local training data, the contribution of the model parameter updates uploaded in this training round to the improvement of the global model performance, and the online training time of the local node.

7. The system according to claim 1, characterized in that, The services provided by the cross-node joint analysis application module include: distributing the trained global model to local data nodes for local health risk prediction; or coordinating the server to aggregate encrypted feature statistics extracted by each node under a unified model for group health trend analysis.

8. The system according to claim 1, characterized in that, The system supports cross-generational health data analysis, and local data nodes can belong to different members of the same family; the federated learning coordination server can train a global model to identify family health risk patterns and return personalized family health collaborative intervention suggestions to each member node.

9. The system according to claim 1, characterized in that, The blockchain evidence storage and auditing module will store at least one of the following information on the blockchain: the version hash value of the global model, the hash of the aggregated result of each round of federated learning, the transaction record of incentive point allocation, and the data usage authorization certificate.

10. A method for multimodal health data fusion and privacy protection based on federated learning, applied to the system as described in any one of claims 1-9, characterized in that, Includes the following steps: S1: The federated learning coordination server initializes the global model and distributes it to the participating multimodal local data nodes; S2: Each local data node uses its local multimodal health raw data to train the global model locally, and obtains local model parameter updates; S3: Each local data node processes the local model parameter updates through the privacy-enhancing computing module and then uploads them to the federated learning coordination server. S4: The federated learning coordination server securely aggregates all received parameter updates to generate a new generation of global model; S5: Repeat steps S2-S4 until the global model converges or the predetermined number of training rounds is reached; S6: Utilize a trained global model or intermediate features to provide health data analysis services through cross-node joint analysis application modules; S7: Throughout the federated learning process, key operations are recorded on the blockchain through the blockchain evidence storage and auditing module.

Images