Privacy protection system for medical data based on federated learning

Abstract

The application discloses a kind of privacy protection systems for medical data based on federal learning, it is related to medical data management technical field, including medical terminal equipment, medical edge server and medical cloud center server;Medical terminal equipment can be preprocessed to medical data by variational modeling to realize privacy enhancement and obtain model training data;Medical edge server is used to pass into multimodal model, screening is obtained vital sign area feature, training is carried out to global medical model, and local model is obtained, local differential perturbation noise is added in local model gradient;Initialization and update global medical model.The application is based on the cloud edge of federal learning, intelligent, safe, trusted architecture, can realize medical model training under cloud edge intelligent cooperation;Realize the fine-grained classification of medical data multimodal fusion, improve the accuracy of model;Variational modeling and differential privacy are integrated into system architecture, ensure the high confidentiality of medical data.

Description

Technical Field

[0001] This invention relates to the field of medical data management technology, and more specifically, to a privacy protection system for medical data based on federated learning. Background Technology

[0002] With the advent of the machine learning era and the rapid development of wearable sensors, the Internet of Things in Healthcare (IoMT) empowers healthcare to achieve intelligent diagnosis and treatment, and the artificial intelligence paradigm has powerfully promoted the development of healthcare. Medical data collection primarily relies on intelligent medical sensors, such as smart bracelets and smart bands. After the sensors collect and preprocess the data, it is transmitted to a cloud computing center for data analysis, forming relevant models to assist in diagnosis and treatment. However, because almost all processing in current IoMT devices is performed in a classical manner (centralized computing) or based on cloud services, we cannot effectively apply machine learning methods within these devices. Specifically, the main problems are as follows:

[0003] 1. Machine learning in cloud computing models suffers from significant communication latency, hindering the effective delivery of data to users. User devices also face insufficient computing resources. These issues necessitate a new distributed model paradigm to address them.

[0004] 2. Due to the significant differences in the datasets uploaded by different medical sensors, obtaining an accurate diagnostic model is quite complex, and the accuracy of the model needs to be improved.

[0005] 3. Medical cloud center servers are untrusted, and communication networks are complex and intertwined. Therefore, sharing medical data carries the risk of privacy leaks.

[0006] To address the aforementioned issues, some scholars have proposed replacing cloud computing architectures with distributed edge computing architectures, offloading model training tasks to medical edge servers to achieve efficient model training. Simultaneously, combining this with federated learning architectures allows data collection and preprocessing to occur only at the sensor end, effectively solving the "data silo" problem between limited sensor computing power and medical data. Kairouz et al. uploaded preprocessed data and used medical edge server resources to train medical models to aid in diagnosis and treatment. This eliminates the need for raw medical data to be transmitted over the network, reducing the risk of data privacy breaches and resolving the issue of sensors being unable to train models. However, with the rise of attacks such as gradient leakage, man-in-the-middle attacks, and collusion attacks, attackers can recover raw medical data using methods like model gradient inference and malicious interception, leading patients to be unwilling to upload their medical data to medical edge servers. Furthermore, the classic FedAvg model aggregation method based on federated learning relies on cloud medical centers, resulting in high communication latency, insufficient bandwidth, and network congestion. While delegating medical data analysis and processing to medical edge servers can reduce the number of data transmissions from medical sensors and shorten communication time, the risk of data leakage remains, necessitating new solutions.

[0007] Zhang et al. proposed a federated learning homomorphic encryption method to address the privacy leakage problem. It homomorphically encrypts data from medical sensors before machine learning. However, homomorphically encrypting the massive amounts of data generated by numerous medical sensors would consume significant time and space, which is nearly impossible for the real-time requirements of medical scenarios. Secondly, the secure multi-party computation proposed by Li et al. is not suitable for distributed medical systems. Wei et al. suggested applying differential privacy to edge computing scenarios by adding noise to medical sensor data to perform differential perturbation. However, adding noise can distort medical data to some extent, reducing model accuracy and consequently lowering the accuracy of medical diagnosis or treatment. Wu et al. proposed an adaptive differential privacy method for edge scenarios, which improves training efficiency by adaptively pruning model gradients, thereby reducing throughput and latency. Yu et al. suggested using the sensitivity of different neural network layers for model compression. This not only solves the problem of redundant weight parameters but also protects data privacy to some extent. Li et al. proposed using parameter sparsity to transmit parameters that are not zero after ANDing with a mask, which effectively prevents model parameter leakage; however, this method reduces model accuracy. Wu et al. proposed using multimodal learning methods for fine-grained filtering to improve accuracy.

[0008] While the above methods can protect patient privacy data to some extent, in edge computing scenarios, federated learning requires significant communication costs and is vulnerable to various attack methods and compromises to model accuracy, as detailed below:

[0009] The surge in the number of smart terminal devices has led to network overload. When a large amount of data is transmitted over the network, it can cause network congestion and affect medical outcomes.

[0010] In medical settings, patients' sensitive data faces issues such as privacy leaks and malicious attacks when transmitted over the network.

[0011] Patients' reluctance to share data and the strictness of data supervision systems have led to a lack of large-scale data sharing, resulting in the "data silo" problem. Summary of the Invention

[0012] The present invention provides a privacy protection system for medical data based on federated learning, which can alleviate the above-mentioned problems.

[0013] To alleviate the above problems, the technical solution adopted by the present invention is as follows:

[0014] A privacy protection system for medical data based on federated learning includes medical terminal devices, medical edge servers, and medical cloud center servers;

[0015] The medical terminal device is used for

[0016] Real-time collection of medical data from patients

[0017] Variational modeling is used to preprocess medical data to enhance privacy and obtain model training data.

[0018] The model training data is transmitted to the medical edge server;

[0019] The medical edge server is used for

[0020] Obtain the global medical model to be trained from the medical cloud center server.

[0021] The training data of the model images is fed into the multimodal model, and the vital sign region features are obtained by filtering.

[0022] A global medical model is trained using vital sign region features to obtain a local model. Local differential perturbation noise is then added to the gradient of the local model.

[0023] The local model gradient, after adding local differential perturbation noise, is sent to the medical cloud center server.

[0024] The medical cloud center server is used for

[0025] An initial global medical model is generated and distributed to the respective medical edge servers to begin the first round of federated learning.

[0026] The system receives and aggregates the local model gradients with added local differential perturbation noise to obtain the global model gradient. Global noise is then added to the global model gradient. The local differential perturbation noise is achieved by perturbing the local gradients of the edge servers using the Laplace mechanism, while the global noise is obtained by perturbing the aggregated gradients using the Gaussian mechanism.

[0027] The global medical model parameters are updated using the gradient of the global model with added global noise to generate a new global medical model. When the new global medical model converges, the federated learning ends; otherwise, the new global medical model is distributed to each of the aforementioned medical edge servers to enter the next round of federated learning.

[0028] In a preferred embodiment of the present invention, the medical terminal device is an intelligent medical sensor.

[0029] In a preferred embodiment of the present invention, the medical cloud center server is also used to monitor the status of the medical edge server cluster, manage the medical terminals, and back up training and diagnostic data.

[0030] In a preferred embodiment of the present invention, the variational modeling process includes:

[0031] Using a fully connected network E, hidden medical data x is obtained to retrieve hidden data z;

[0032] The hidden data z is input into the probabilistic encoder B to obtain an approximate probability distribution b;

[0033] The approximate probability distribution b is input into the decoder D to obtain the model training data.

[0034] In a preferred embodiment of the present invention, the medical edge server uses a multimodal model to capture the most discriminative regions in the model training data and removes background noise from the model training data to obtain vital sign region features.

[0035] In a preferred embodiment of the present invention, the process of obtaining vital sign region features through multimodal model screening includes the following steps:

[0036] S1. Encode the two modalities of the model training data separately to obtain the encoded features e of the two modalities. a ,e v ;

[0037] S2, Encode the features e of the two modalities a ,e v Multimodal cross-decoding is performed to obtain fused features;

[0038] S3. Input the fused features into the fine-grained filtering module to filter out the vital signs region features.

[0039] In a preferred embodiment of the present invention, in step S1, before encoding, the model training data needs to be processed by multimodal processing through convolutional pooling, and then encoded. The encoding formula is as follows:

[0040] v1, a1 = Conv2d(v0, a0)

[0041] v2, a2 ​​= MaxPooling(v1, a1)

[0042] e v =E v ([cls v v2]+pos v )

[0043] e a =B a ([cls a a2]+pos a )

[0044] Where v0 and a0 represent the features of the two modalities in the model training data, v1 and a1 represent the features of the modalities after passing through a 2D convolutional layer, v2 and a2 represent the features of the modalities after convolutional max pooling, and e a e v E represents the encoded features of two modes. v and E a They have the same structure, representing the encoding modules for two different modes; cls v and pos v Let cls represent the learnable classification embedding and location embedding of the two modalities, respectively. a and pos a Similarly.

[0045] In a preferred embodiment of the present invention, step S3 specifically includes the following steps:

[0046] S31. The total attention weights of the L layers for the two modalities corresponding to the multimodal cross-decoding network are expressed as follows:

[0047]

[0048]

[0049] S32. Multiply the attention weights of the two modalities in each layer of the multimodal cross-decoding network, and the weights of the l-th layer of the multimodal cross-decoding network are expressed as follows:

[0050]

[0051] The total weights of layer L in a multimodal cross-decoding network are then represented as W = [w 0 w1 ,…,w L-1 Assuming there are N patches in total, the weights of the l-th layer of the multimodal cross-decoding network are expressed as follows:

[0052]

[0053] For each patch, select the largest weight from the L layer of the multimodal cross-decoding network to form a new weight matrix w. max :

[0054] w max =[w′0,w′1,…,w′ N ]

[0055] Where w′ i (i = 0, 1, ..., N) represents the maximum weight of the i-th patch in layer L;

[0056] S33. According to the weight matrix w max Given the index of the largest weight and the number of patches to be selected, n, select the n largest weight indices j = [j0, j1, ..., jn]. n ], and according to the subscript j, the fusion feature O L-1 Screening was performed to obtain the characteristics of vital sign regions.

[0057]

[0058] In a preferred embodiment of the present invention

[0059] Suppose the medical edge server trains the global medical model in the t-th round. Then, in this round of training, the model gradient needs to be descent, as shown in the formula:

[0060]

[0061] Next, the model gradient is clipped using the following formula:

[0062]

[0063] Update the model parameters again, using the following formula:

[0064] ω i (t+1)=ω i (t)-η*g i (t);

[0065] Once the medical edge server completes training the global medical model, i.e., reaches the threshold for the number of training rounds of the local model in this federated training, the local model is obtained. The formula for adding local differential perturbation noise to the gradient of the local model is:

[0066]

[0067] Where lap is the Laplace distribution, Δg is the query sensitivity, and h(.) represents the normalization of the model gradient.

[0068] Compared with the prior art, the beneficial effects of the present invention are:

[0069] A distributed architecture for medical data based on cloud-edge-device is proposed. This architecture is based on federated learning and is characterized by being cloud-edge, intelligent, secure, and trustworthy, enabling the training of medical models under intelligent cloud-edge collaboration.

[0070] By using multimodal model screening to obtain vital sign region features for model training, fine-grained classification of multimodal fusion of medical data was achieved. By extracting key regions of medical data as input for model training, the accuracy of the model was improved.

[0071] Integrating variational modeling and differential privacy into the system architecture can ensure the high confidentiality of medical data.

[0072] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, embodiments of the present invention are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0073] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is an architecture diagram of the system of the present invention;

[0075] Figure 2 This is a diagram illustrating the system data processing and flow of the present invention;

[0076] Figure 3 This is a flowchart of the system method (PPHFL) of the present invention;

[0077] Figure 4 This is a flowchart of the variational modeling process of the present invention;

[0078] Figure 5 This is a flowchart of the multimodal fine-grained image processing of the present invention;

[0079] Figure 6 This is a schematic diagram of differential privacy processing for the medical edge server in the system of the present invention;

[0080] Figure 7 This is a comparison chart of the accuracy rates of the three schemes during the experiment of this invention;

[0081] Figure 8 This is a comparison chart of the loss values ​​of the three schemes during the experiment of this invention;

[0082] Figure 9 These are defense effect diagrams of PPHFL on different models during the experimental process of this invention;

[0083] Figure 10 This is a gradient leakage diagram illustrating the experimental process of this invention;

[0084] Figure 11 This is an example of gradient leakage during the experimental process of this invention. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0086] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0087] Please refer to Figure 1 This invention discloses a privacy protection system for medical data based on federated learning, comprising three layers of entities, from bottom to top: medical terminal equipment, medical edge server, and medical cloud center server.

[0088] Medical terminal equipment consists of medical sensors, which are responsible for collecting medical data from patients in real time.

[0089] Medical terminal devices form the layer closest to the patient, and their number depends on the individual patient needs and the overall patient population. Depending on the specific medical terminal device, different technologies can be used to acquire and manage medical data, providing data support for training auxiliary diagnostic models. Furthermore, these medical terminal devices are intelligent devices with specific data preprocessing or computational functions. They can preprocess medical data through variational modeling to enhance privacy and obtain model training data, as well as transmit model training data to medical edge servers.

[0090] Medical edge servers are typically located in hospital departments or scattered local hospitals, such as community hospitals and cardiology hospitals, distributed across multiple locations to provide services to residents within the region. Compared to medical terminal devices, medical edge servers typically have enormous memory and computing power. During model training, the medical edge server is primarily responsible for retrieving the global medical model to be trained from the medical cloud center server, inputting the model training data into the multimodal model, filtering out vital sign region features, using the vital sign region features to train the global medical model to obtain a local model, adding local differential perturbation noise to the local model gradient, and sending the local model gradient with added local differential perturbation noise to the medical cloud center server.

[0091] During the assisted diagnosis process, the medical edge server is responsible for calculating the diagnostic results, which are then fed back to the user and the cloud to provide low-latency computing services.

[0092] The medical cloud center server is a centralized medical platform primarily responsible for deploying federated learning applications and issuing model training tasks. This includes distributing the global medical model to be trained and collecting and aggregating local model gradients from various hospital edge servers, incorporating local differential perturbation noise. Furthermore, it can monitor the status of the edge server cluster and manage and schedule resources through real-time monitoring. Compared to medical edge servers, it can dynamically adjust resources and node selection, such as selecting suitable edge servers for training based on their computing power, memory, and datasets. It also boasts greater computing power and memory. However, its distance from patients presents challenges such as high communication latency and high energy consumption when interacting directly with them.

[0093] Please refer to Figure 2 and Figure 3 The present invention, employing federated learning, implements a privacy protection system for medical data. This system enables federated training of medical models, using variational modeling at medical device terminals and differential privacy mechanisms at medical edge servers and medical cloud center servers, thereby achieving the goal of privacy protection. The specific process is as follows:

[0094] 1. Medical terminal equipment collects patients' medical data in real time.

[0095] 2. Medical terminal devices use variational modeling to preprocess medical data x to enhance privacy and obtain model training data. Then it is uploaded to the medical edge server.

[0096] To prevent gradient leakage attacks on raw data, we protect privacy by performing variational processing on the data.

[0097] A privacy enhancement module for medical data is implemented using a variational bottleneck. The variational bottleneck consists of a probabilistic encoder B and a decoder D.

[0098] We use a fully connected network E to hide the medical data x, obtaining hidden data z = E(x). Then, we input the hidden data z into the probabilistic encoder B to obtain an approximate probability distribution b. We then feed the approximate probability distribution b into the decoder D to obtain the model training data. Where b = q(B|z), q represents the probabilistic encoding function, p represents the decoding function, and the variational modeling process is as follows: Figure 4 As shown.

[0099] Specifically, the probabilistic encoder B is a fully connected layer of the fully connected network E. Assuming the medical data x has a dimension of k, then the probabilistic encoder B has 2k neurons. The medical data x is used as input to the fully connected network E, and we compute the representation z of the fully connected network E. Then, z is combined with the weights and biases of the neurons in the probabilistic encoder B to compute an approximate probability distribution b. The approximate probability distribution b is a vector with the same dimension as z. We feed z into the decoder D to obtain the model training data. The decoder D contains k neurons. The weights of the neurons are {1,2,3,...,|z|}. This is the result of b multiplied by the corresponding neuron weight. Finally, we use the model training data... As input to O, where O corresponds to the neural network layer after the strain divider bottleneck, i.e., the medical network model on the medical edge server.

[0100] 3. The medical cloud center server initializes a global medical model that needs to be trained and distributes it to the medical edge server.

[0101] 4. The medical edge server inputs the model training data into the multimodal model and filters out the vital signs region features.

[0102] Since differential perturbation mechanisms can lead to a decrease in model accuracy, but medical image data requires extremely high accuracy, we designed a multimodal fine-grained auxiliary diagnostic model to improve model accuracy. Our approach involves inserting a multimodal model before the input layer of the medical model to capture the most discriminative regions in the medical image data and eliminate background noise from the model training data, thereby obtaining vital sign region features. The model algorithm flow is as follows: Figure 5 As shown, the specific steps include:

[0103] S1. Encode the two modalities of the model training data separately to obtain the encoded features e of the two modalities. a e v .

[0104] Before encoding, the model training data needs to be processed using a convolutional pool for multimodal processing, and then encoded. The encoding formula is as follows:

[0105] v1, a1 = Conv2d(v0, a0)

[0106] v2, a2 ​​= MaxPooling(v1, a1)

[0107] e v =E v ([cls v v2]+pos v )

[0108] e a =E a ([cls a a2]+pos a )

[0109] Where v0 and a0 represent the features of the two modalities in the model training data, v1 and a1 represent the features of the modalities after passing through a 2D convolutional layer, v2 and a2 represent the features of the modalities after convolutional max pooling, and e a e v This represents the coding features of two modalities. E v and E a They have the same structure, representing the encoding modules for two different modes; cls v and pos v Let cls represent the learnable classification embedding and location embedding of the two modalities, respectively. a and pos a Similarly.

[0110] S2, Encode the features e of the two modalities a e v Multimodal cross-fusion and cross-decoding are performed to obtain the fused features.

[0111] S3. Input the fused features into the fine-grained filtering module to filter and obtain vital sign region features, specifically including the following steps:

[0112] S31. The total attention weights of the L layers for the two modalities corresponding to the multimodal cross-decoding network are expressed as follows:

[0113]

[0114]

[0115] S32. Multiply the attention weights of the two modalities in each layer of the multimodal cross-decoding network, and the weights of the l-th layer of the multimodal cross-decoding network are expressed as follows:

[0116]

[0117] The total weights of layer L in a multimodal cross-decoding network are then represented as W = [w 0 w 1 ,…,w L-1 Assuming there are N patches in total, the weights of the l-th layer of the multimodal cross-decoding network are expressed as follows:

[0118]

[0119] For each patch, select the largest weight from the L layer of the multimodal cross-decoding network to form a new weight matrix w. max :

[0120] w max =[w′0,w′1,…,w′ N ]

[0121] Where w′ i (i = 0, 1, ..., N) represents the maximum weight of the i-th patch in layer L;

[0122] S33, w max This refers to the selection criteria we retrieved from the attention weights. By fusing the weights of the two modalities and selecting the maximum weight for each patch across all layers, we obtain the key discriminant regions that both modalities commonly focus on during training. Specifically, the fused features output after obtaining the L-layer multimodal decoding are... Then, based on the weight matrix w max Given the index of the largest weight and the number of patches to be selected, n, select the n largest weight indices j = [j0, j1, ..., jn]. n ], and according to the subscript j, the fusion feature O L-1 Screening was performed to obtain the characteristics of vital sign regions.

[0123]

[0124] 5. Train the global medical model using vital sign region features to obtain a local model, and then add local differential perturbation noise to the gradient of the local model, such as... Figure 6 As shown.

[0125] Let the medical edge server S be... i The vital signs region characteristics are D i D i =(X i Y i Let the model output be Y, then:

[0126] Y = f(X) i )

[0127] Comparison of Y and Y i The accuracy and loss values ​​are calculated. After each training round, the test dataset is input into the currently trained global medical model to calculate the accuracy and loss values. When the threshold of the number of training rounds is reached, the training of the global medical model is completed, and a local model is obtained. Then, local differential perturbation noise is added to the gradient of the local model and uploaded to the medical cloud center server.

[0128] Suppose that the medical edge server trains the current global medical model in round t. Then, in this round of training, the model gradient needs to be descented, as shown in the formula:

[0129]

[0130] Then the model gradient g i (t) is used for cutting, and the formula is:

[0131]

[0132] Update the model parameters ω again i The formula is as follows:

[0133] ω i (t+1)=ω i (t)-η*g i (t);

[0134] Once the medical edge server has completed training the global medical model, that is, after reaching the threshold for the number of training rounds of the local model in this federated training, the local model is obtained. The formula for adding local differential perturbation noise to the gradient of the local model is as follows:

[0135]

[0136] Where lap is a Laplace distribution, Δg i For query sensitivity, h(.) represents the normalization of the model gradient.

[0137] 6. The medical cloud center server receives and aggregates various locally added differential perturbation noises. Local model gradient Obtain the global model gradient and add global noise to it. The set of local model gradients from different edge servers is ΔM

[0138]

[0139] Compute its edge server S i Paradigm Obtain the general normal form Calculate sensitivity S f The median of ζ is added to the global model noise. Where σ is the Gaussian parameter and N is the Gaussian distribution.

[0140] 7. The medical cloud center server uses the gradient of the global model after adding global noise to update the parameters of the global medical model and generate a new global medical model. When the new global medical model converges, the federated learning ends; otherwise, the new global medical model is distributed to each of the aforementioned medical edge servers to enter the next round of federated learning.

[0141] To verify whether PPHFL can balance privacy and accuracy, we conducted the following simulation experiment.

[0142] I. Experimental Setup

[0143] The experiment used Alibaba Cloud servers as cloud data centers, two Raspberry Pis as two hospital edge servers, and a laptop as an edge gateway for data preprocessing. Two virtual machines were used to open multiple ports to simulate multiple edge gateways participating in the training.

[0144] In terms of environment, we used PyTorch for simulation experiments. First, we tested the ResNet18 residual model using the CIFAR10 dataset. Second, we compared it with FedAvg and differential privacy schemes without multi-modal fusion. Finally, PPHFL outperformed FedAvg in attack success rates across different model architectures, demonstrating its resistance to privacy-leakage attacks.

[0145] II. Accuracy Analysis

[0146] To verify the accuracy of PPHFL in image recognition, we conducted comparative experiments on the CIFAR10 dataset using the Resenet18 residual network. CIFAR10 is a classic image dataset containing 10 object classes. To make the results more representative, we compared the classic federated aggregation method FedAvg with a federated aggregation scheme that incorporates differential privacy mechanisms. The accuracy comparison is shown in the figure below. Figure 7 As shown in the comparison chart of loss values, Figure 8 As shown.

[0147] from Figure 7As can be seen, FedAvg, as a classic federated average aggregation scheme, achieves an accuracy of 86% upon convergence, PPHFL achieves 86.63%, and DP-Fed achieves 84%. Specifically, PPHFL, due to the addition of noise on the terminal device, affects model accuracy to some extent, but it still achieves good accuracy in the ResNet18 residual network, slightly lower than FedAvg but significantly higher than DP-Fed. At the beginning of training, due to the presence of noise, PPHFL's accuracy is only around 20%. After about 10 rounds of training, the accuracy reaches around 75%, and then slowly improves. Figure 7 As can be seen, PPHFL's stability is significantly higher than DP-Fed and roughly on par with FedAvg, indicating that multimodal fusion can improve model accuracy.

[0148] from Figure 8 It was also observed that in the first few rounds of training, the loss value of PPHFL was on par with DP-Fed. After about 10 rounds of training, the loss value decreased to below 1. When the number of training rounds reached 30, the loss value of PPHFL tended to stabilize at around 0.48, roughly the same as FedAvg, while DP-Fed only approached stability after nearly 50 rounds, and was higher than both FedAvg and PPHFL. This shows that PPHFL can achieve a high accuracy rate for image recognition while protecting privacy.

[0149] In addition, we conducted experiments on the CIFAR10 and CIFAR100 datasets on ResNet8, ResNet20, ShuffleV1, and Vgg8 models, comparing them with FedAvg, and obtained the accuracy shown in Tables 1 and 2.

[0150] Table 1. Accuracy Comparison on the CIFAR10 Dataset

[0151] Method Resnet8 Resnet20 ShuffleV1 Vgg8 PPHFL 78.16 75.48 74.22 77.14 FedAvg 77.48 74.88 73.92 77.06

[0152] Table 2 shows the accuracy comparison on the CIFAR100 dataset.

[0153] Method Resnet8 Resnet20 ShuffleV1 Vgg8 PPHFL 40.64 40.42 39.37 31.02 FedAvg 40.33 40.28 39.25 30.83

[0154] As can be seen from Tables 1 and 2, PPHFL has a higher accuracy than FedAvg on all models, indicating that even under the influence of differential noise, fine-grained screening of key regions can help improve the accuracy of the model. Therefore, PPHFL is well adapted to medical diagnosis and treatment scenarios that require high accuracy.

[0155] III. Efficiency Analysis

[0156] To test the efficiency of PPHFL, we also compared FedAvg and PPHFL, training different schemes on the same device. The training time comparison is shown in Table 3.

[0157] Table 3 Comparison of Training Time for the Three Schemes

[0158] plan PPHFL FedAvg DP-Fed Average time per round (s) 447.95 431.08 468 Average convergence time (s) 21950 21123 23400

[0159] As shown in Table 3, the training time of PPHFL is higher than that of FedAvg but lower than that of DP-Fed. Compared to FedAvg, PPHFL requires homomorphic encryption and decryption operations on model parameters, which leads to an increase in training time. However, the time taken for PPHFL to converge the model is not much different from that of FedAvg and is much lower than that of DP-Fed. This indicates that PPHFL can be applied to real-time medical scenarios that require privacy protection.

[0160] IV. Privacy Analysis

[0161] We used three progressively more complex baseline model architectures: Shallow MLP (SMLP), Deep MLP (DMLP), and LeNet, to test Attack Success Rate (ASR). Reconstruction quality was measured by calculating the mean squared error (MSE) and structural similarity (SSIM) between the original and reconstructed images. Lower MSE and higher SSIM values ​​indicate better reconstruction, higher similarity to the original image, and thus better image restoration and higher ASR. Table 4 shows the MSE and SSIM values ​​for different models, and the defensive performance across different model architectures is as follows: Figure 9 As shown.

[0162] Table 4 Comparison of MSE and SSIM under different models

[0163] Model SMLP DMLP LeNet MSE 3.95 4.62 2.07 SSIM 0.03 0.01 0.01

[0164] As shown in Table 4, PPHFL has a high MSE in all three models, but a low SSIM, almost zero. Figure 9 It can be seen that PPHFL has a low ASR, indicating that PPHFL is effective in resisting attacks. Furthermore, we used a gradient leakage attack for simulation testing, simultaneously applying gradient leakage attacks to both PPHFL and FedAvg, and using changes in lung medical images as a comparison. Figure 10 and Figure 11 .

[0165] In gradient leakage attacks targeting FedAvg, attackers first use fake data and labels to train the neural network, and then infer the real training dataset using the resulting gradients. For an N-dimensional gradient vector, the attacker needs at most N+1 attempts to infer the real data. Figure 10 As can be seen, under a brute-force attack, with the increase of iterations, the feature vectors of the image are inferred, and the image is gradually reconstructed. In PPHFL, as... Figure 11 As shown, due to differential privacy and homomorphic encryption mechanisms, the gradients obtained by attackers are encrypted or have noise added, making it impossible to obtain the original image from the gradients. Therefore, PPHFL can effectively resist gradient leakage attacks and protect patients' privacy data.

[0166] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A privacy protection system for medical data based on federated learning, characterized in that, This includes medical terminal equipment, medical edge servers, and medical cloud center servers; The medical terminal device is used for Real-time acquisition of medical image data from patients, Variational modeling is used to preprocess medical data to enhance privacy and obtain model training data. The model training data is transmitted to the medical edge server; The medical edge server is used for Obtain the global medical model to be trained from the medical cloud center server. The training data is fed into the multimodal model, and the vital sign region features are obtained through filtering. A global medical model is trained using vital sign region features to obtain a local model. Local differential perturbation noise is then added to the gradient of the local model. The local model gradient, incorporating local differential perturbation noise, is sent to the medical cloud center server. The medical cloud center server is used for An initial global medical model is generated and distributed to the respective medical edge servers to begin the first round of federated learning. The system receives and aggregates the local model gradients with added local differential perturbation noise to obtain the global model gradient, and then adds global noise to the global model gradient. The global medical model parameters are updated using the gradient of the global model with added global noise to generate a new global medical model. When the new global medical model converges, the federated learning ends; otherwise, the new global medical model is distributed to each of the aforementioned medical edge servers to enter the next round of federated learning. The process of variational modeling includes: Using a fully connected network E, hidden medical data x is obtained to retrieve hidden data z; The hidden data z is input into the probabilistic encoder B to obtain an approximate probability distribution b; The approximate probability distribution b is input into the decoder D to obtain the model training data; The medical edge server uses a multimodal model to capture the most discriminative regions in the model training data and eliminates background noise in the model training data to obtain vital sign region features. The process of obtaining vital sign region features through multimodal model screening includes the following steps: S1. Encode the two modalities of the model training data separately to obtain the encoded features of the two modalities. ; S2. Encode the features of the two modalities. Multimodal cross-decoding is performed to obtain fused features; S3. Input the fused features into the fine-grained filtering module to filter and obtain vital sign region features; Step S3 specifically includes the following steps: S31. The total attention weights of the L layers for the two modalities corresponding to the multimodal cross-decoding network are expressed as follows: ； S32. Multiply the attention weights of the two modalities in each layer of the multimodal cross-decoding network, and the weights of the l-th layer of the multimodal cross-decoding network are expressed as follows: ， The total weights of layer L in the multimodal cross-decoding network are then expressed as: Assuming there are N patches in total, the weights of the l-th layer of the multimodal cross-decoding network are expressed as follows: For each patch, a new weight matrix is ​​formed by selecting the largest weight from the L layer of the multimodal cross-decoding network. : in (i=0,1,…,N) represents the maximum weight of the i-th patch in layer L; S33. According to the weight matrix Given the index of the largest weight and the number of patches to be selected (n), select the n indexes with the largest weights. And based on the subscript j, the fusion features Screening was performed to obtain the characteristics of vital sign regions.

2. The system according to claim 1, characterized in that, The medical terminal device is an intelligent medical sensor.

3. The system according to claim 1, characterized in that, The medical cloud center server is also used to monitor the status of the medical edge server cluster, manage the medical terminal devices, and back up training and diagnostic data.

4. The system according to claim 1, characterized in that, In step S1, before encoding, the model training data needs to be processed using convolutional pooling for multimodal processing, and then encoded. The encoding formula is as follows: in, This represents the features of two modalities in the model training data; This represents the features of a mode after passing through a 2D convolutional layer; This represents the features of a mode after convolutional max pooling; and Indicates the encoding module; and This represents a learnable classification embedding; and This indicates positional embedding.

Images