A medical data privacy protection method based on federated learning and BERT model

Abstract

The present application relates to a medical data privacy protection method based on federated learning and a BERT model, comprising: each medical institution trains a BERT model issued by a central server based on its own medical data set, calculates the corresponding model gradient, and calculates the parameter update amount of the corresponding medical institution based on the perturbed model gradient; each medical institution randomly divides its own parameter update amount, and divides the divided data into levels according to the degree of privacy importance, encrypts the data of each level using a different encryption algorithm to obtain corresponding encrypted data; performs threat detection on the different encrypted data of each medical institution, uploads the threat-free encrypted data to the central server for decryption and data aggregation, and obtains global model parameters; each medical institution updates its own local model parameters based on the global model parameters issued by the central server, and trains the local model based on the updated local model parameters.

Description

Technical Field

[0001] The present application relates to the technical field of medical data privacy protection, and in particular to a medical data privacy protection method based on federated learning and BERT model. Background Art

[0002] Traditional data privacy protection methods, such as data encryption and access control, can provide some security, but they have significant limitations when it comes to medical data sharing. In multi-institutional medical research, institutions need to share data to train more accurate models. However, directly transmitting encrypted data incurs significant communication overhead and computational effort on the receiving end, while frequent decryption increases the risk of data leakage. Currently, the application of federated learning and the BERT model to medical data privacy protection is still in its developmental stages, and existing methods have shortcomings in terms of model training efficiency, privacy protection strength, and data compatibility. Summary of the Invention

[0003] Based on this, it is necessary to provide a medical data privacy protection method based on federated learning and BERT model, which includes:

[0004] S1: Preprocess the local data of each medical institution to obtain the medical data set of each medical institution;

[0005] S2: Each medical institution trains the BERT model sent from the central server based on its own medical dataset, calculates the corresponding model gradient, and perturbs each model gradient. Based on the perturbed model gradient, the parameter update amount for the corresponding medical institution is calculated;

[0006] S3: Each medical institution randomly divides its own parameter update amount and classifies the divided data into levels according to the privacy importance. Different encryption algorithms are used to encrypt the data of each level to obtain the corresponding encrypted data;

[0007] S4: Perform threat detection on each medical institution's encrypted data, discard any threatening encrypted data, and re-execute steps S3-S4; upload the non-threatening encrypted data to the central server for decryption and data aggregation to obtain global model parameters;

[0008] S5: Each medical institution updates its local model parameters based on the global model parameters issued by the central server, and trains the local model based on the updated local model parameters.

[0009] Preferably, preprocessing the local data of any medical institution includes:

[0010] The local data includes text data describing the patient's basic information and the disease the patient suffers from;

[0011] Clean the local data of each medical institution separately, remove duplicate, missing, and non-compliant data, and fill in the data for the removed parts;

[0012] Based on the disease types of patients in each local data, the professional classification of medical institutions is used to label the corresponding local data to obtain the disease department label corresponding to each local data;

[0013] Use a specialized medical word segmenter to segment each filled local data, and convert each segmented local data into the vector format required by the BERT model.

[0014] The transformed vectors are used as data samples, and the disease department labels corresponding to the local data are used as labels for the data samples to construct the medical data sets corresponding to the medical institutions.

[0015] Preferably, before the word segmentation processing of the local data, dynamic data generalization is also performed on the padded local data:

[0016] Generalize the specific age of patients in the filled local data into age ranges;

[0017] The specific disease type of the patient in the filled local data is generalized into chronic disease / acute disease.

[0018] Preferably, the central server sends the BERT model including:

[0019] Obtain a large-scale professional corpus in the medical field, which includes a plurality of medical data samples and their corresponding first true labels; the content of the medical data sample corresponds to the content of the data sample, and the type of the first true label corresponds to the type of the label of the data sample;

[0020] Input each medical data sample into the BERT model and output the corresponding first predicted label;

[0021] Calculating a first cross entropy loss based on any first predicted label and the corresponding first true label, and adding a regularization term to the first cross entropy loss to obtain a loss function corresponding to the medical data sample;

[0022] Sum the loss functions of all medical data samples to get the total loss, minimize the total loss, and get the pre-trained BERT model;

[0023] Initialize the pre-trained BERT model and distribute the model parameters of the initialized BERT model to various medical institutions.

[0024] Preferably, any medical institution trains the BERT model issued by the central server based on the corresponding medical data set, and calculates the corresponding model gradient including:

[0025] Receive the model parameters of the initialized BERT model as the initial local model parameters of the corresponding medical institution;

[0026] In the corresponding medical dataset, the label of each data sample is used as the second true label, and each data sample is input into the pre-trained BERT, which outputs the corresponding second predicted label;

[0027] Calculate a second cross entropy loss based on any second predicted label and the corresponding second true label, and multiply the second cross entropy loss by the gradient operator to obtain the gradient of the second cross entropy loss of the corresponding data sample to the local model parameter;

[0028] The gradients of the second cross entropy loss of all data samples with respect to the local model parameters are averaged to obtain the model gradients of the second cross entropy loss of the corresponding medical dataset with respect to the local model parameters.

[0029] Preferably, perturbing the model gradient, and calculating the parameter update amount of the corresponding medical institution based on the perturbed model gradient includes:

[0030] Add noise that obeys the Laplace distribution to the model gradient to obtain the perturbed model gradient;

[0031] Multiply the perturbed model gradient by the learning rate to get the first product;

[0032] The negative value of the first product is used as the parameter update amount of the corresponding medical institution.

[0033] Preferably, S3 includes:

[0034] Any medical institution randomly divides its parameter update amount into several sub-data;

[0035] Based on all elements in any sub-data, calculate the L2 norm of the corresponding sub-data; and calculate the mean and standard deviation of the L2 norm based on the L2 norm of all sub-data;

[0036] When the L2 norm of any sub-data is greater than or equal to the sum of the mean and standard deviation of the L2 norm, the privacy importance of the corresponding sub-data is determined to be high, and the sub-data with high privacy importance is encrypted using a symmetric encryption algorithm to obtain first encrypted data;

[0037] When the L2 norm of any sub-data is greater than the difference between the mean and standard deviation of the L2 norm, and less than the sum of the mean and standard deviation of the L2 norm, the privacy importance of the corresponding sub-data is determined to be medium. The sub-data with the medium level is encrypted using an asymmetric encryption algorithm to obtain the second encrypted data;

[0038] When the L2 norm of any sub-data is less than or equal to the difference between the mean and standard deviation of the L2 norm, the privacy importance of the corresponding sub-data is determined to be low. The sub-data with a low level is encrypted using a homomorphic encryption algorithm to obtain third encrypted data.

[0039] Preferably, the non-threatening encrypted data is uploaded to the central server for decryption and data aggregation, and the global model parameters obtained include:

[0040] After any threat-free encrypted data is uploaded to the central server, it is decrypted using a symmetric decryption algorithm, an asymmetric decryption algorithm, or a homomorphic decryption algorithm to obtain the corresponding decrypted data. When the decryption time exceeds the preset maximum decryption time, the corresponding encrypted data is marked as lost and threat detection is performed again.

[0041] Recombining the decrypted data belonging to the same medical institution into a second parameter update amount corresponding to the medical institution;

[0042] The mean of the second parameter update amounts of all medical institutions is calculated, and the global model parameters of this round of iteration are updated with the mean of the second parameter update amounts to obtain the global model parameters of the next round of iteration.

[0043] Preferably, the preset maximum decryption time includes: maximum decryption time for low-level data / maximum decryption time for intermediate-level data / maximum decryption time for high-level data;

[0044] The maximum decryption time for low-level data is the maximum decryption time of the homomorphic decryption algorithm;

[0045] The maximum decryption time for intermediate data is the maximum decryption time of the asymmetric decryption algorithm;

[0046] The maximum decryption time for advanced data is the maximum decryption time of the symmetric decryption algorithm.

[0047] Preferably, key identification is also included before decryption:

[0048] Obtain the name, geographic location, environmental characteristics, equipment information, and timestamp of each medical institution;

[0049] Combine the name, geographic location, equipment information, and timestamp of any medical institution into a key seed;

[0050] The key seed and the corresponding environmental characteristics are passed through a key derivation function or a random number generator to generate a key;

[0051] The key corresponding to each medical institution is uploaded to the central server together with the non-threatening encrypted data. The central server traces the source based on the key carried by each non-threatening encrypted data, queries the corresponding medical institution, and determines whether the corresponding medical institution is legal. The non-threatening encrypted data uploaded by the legal medical institution is decrypted and the non-threatening encrypted data uploaded by the illegal medical institution is discarded.

[0052] Beneficial effects: This method enables the sharing of medical data while protecting patient privacy information. At the same time, it reduces the communication overhead and computing power of each medical institution. Each medical institution updates its own local model parameters based on the issued global model parameters, and trains the local model of the corresponding medical institution based on the updated local model parameters. The trained local model can finally be applied to the analysis and prediction tasks of local data, further improving treatment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 This is a flowchart of the medical data privacy protection method based on federated learning and BERT model in an embodiment of this application. DETAILED DESCRIPTION

[0055] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0056] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0057] like Figure 1 As shown, this embodiment provides a medical data privacy protection method based on federated learning and BERT model, which includes:

[0058] S1: Preprocess the local data of each medical institution to obtain the medical data set of each medical institution.

[0059] Specifically, preprocessing the local data of any medical institution includes:

[0060] The local data includes text data describing the patient's basic information and the disease the patient suffers from;

[0061] Clean the local data of each medical institution separately, remove duplicate, missing, and non-compliant data, and fill in the data for the removed parts;

[0062] In this embodiment, the text data corresponding to similar diseases are used to fill the eliminated areas. If there are multiple similar diseases, the majority of the text data corresponding to these diseases is used to fill the eliminated areas.

[0063] Based on the disease types of patients in each local data, the professional classification of medical institutions is used to label the corresponding local data to obtain the disease department label corresponding to each local data;

[0064] Use a specialized medical tokenizer (such as Jieba) to tokenize each padded local data, and convert each tokenized local data into the vector format required by the BERT model (including a combination of word vectors, position vectors, and segment vectors).

[0065] Word segmentation example: split "the patient suffers from diabetes" into "the patient suffers from diabetes";

[0066] The transformed vectors are used as data samples, and the disease department labels corresponding to the local data are used as labels for the data samples to construct the medical data sets corresponding to the medical institutions.

[0067] In this embodiment, before the word segmentation processing of the local data, dynamic data generalization is also performed on the padded local data to protect the patient's privacy:

[0068] Generalize the specific age of patients in the filled local data into age ranges;

[0069] The specific disease type of the patient in the filled local data is generalized into chronic disease / acute disease.

[0070] Data generalization examples: 1. Generalize "age 42 years old" to "age 40-45 years old"; 2. Generalize "diabetes" to "chronic disease".

[0071] S2: Each medical institution trains the BERT model sent by the central server based on its own medical data set, calculates the corresponding model gradient, and perturbs each model gradient. Based on the perturbed model gradient, the parameter update amount of the corresponding medical institution is calculated.

[0072] Specifically, the BERT model sent by the central server includes:

[0073] Obtain a large-scale professional corpus in the medical field, which includes a plurality of medical data samples and their corresponding first true labels; the content of the medical data sample corresponds to the content of the data sample, and the type of the first true label corresponds to the type of the label of the data sample;

[0074] Input each medical data sample into the BERT model and output the corresponding first predicted label;

[0075] Calculating a first cross entropy loss based on any first predicted label and the corresponding first true label, and adding a regularization term to the first cross entropy loss to obtain a loss function corresponding to the medical data sample;

[0076] Sum the loss functions of all medical data samples to get the total loss, minimize the total loss, and get the pre-trained BERT model;

[0077] Initialize the pre-trained BERT model and distribute the model parameters (such as the number of layers, hidden layer dimensions, etc.) of the initialized BERT model to various medical institutions.

[0078] In this embodiment, the BERT model is pre-trained and initialized on the central server only in the first round of iteration. The global model parameters in the first round of iteration are the model parameters of the initialized BERT model. Subsequent iterations use the global model parameters obtained by the update of the previous round of iteration for federated learning. In the subsequent federated learning process, through continuous iterative updates, the global model parameters are gradually optimized to be more suitable for the local data of each medical institution.

[0079] Regularization terms are added to prevent model overfitting. At the same time, the word embedding layer of the BERT model is optimized based on the large amount of professional terminology in a large-scale medical professional corpus, and a domain-adaptive word vector training method is adopted to enable the model to better understand the semantics of medical vocabulary.

[0080] Furthermore, any medical institution trains the BERT model issued by the central server based on the corresponding medical dataset and calculates the corresponding model gradients including:

[0081] Receive the model parameters of the initialized BERT model as the initial local model parameters of the corresponding medical institution;

[0082] In the corresponding medical dataset, the label of each data sample is used as the second true label, and each data sample is input into the pre-trained BERT, which outputs the corresponding second predicted label;

[0083] Calculate a second cross entropy loss based on any second predicted label and the corresponding second true label, and multiply the second cross entropy loss by the gradient operator to obtain the gradient of the second cross entropy loss of the corresponding data sample to the local model parameter;

[0084] The gradients of the second cross entropy loss of all data samples with respect to the local model parameters are averaged to obtain the model gradients of the second cross entropy loss of the corresponding medical dataset with respect to the local model parameters.

[0085] Furthermore, the model gradient is perturbed, and the parameter update of the corresponding medical institution is calculated based on the perturbed model gradient, including:

[0086] Add noise that obeys the Laplace distribution to the model gradient to obtain the perturbed model gradient; where, s ~ Lap ( D f / d ) represents noise s Follows Laplace distribution Lap (·); Δf is sensitivity, which represents the maximum change in the objective function value when adding or deleting a data sample on the medical dataset; d The privacy budget is used to measure the strength of privacy protection and needs to be set reasonably according to the actual application situation;

[0087] Multiply the perturbed model gradient by the learning rate to get the first product;

[0088] The negative value of the first product is used as the parameter update amount of the corresponding medical institution.

[0089] On the basis of data generalization, differential privacy is used to perturb the model gradient to ensure data privacy.

[0090] S3: Each medical institution randomly divides its own parameter update amount and divides the divided data into levels according to the importance of privacy. Different encryption algorithms are used to encrypt the data of each level to obtain the corresponding encrypted data.

[0091] Specifically, this step includes:

[0092] Any medical institution randomly divides its parameter update amount into several sub-data;

[0093] Based on all elements in any sub-data, calculate the L2 norm of the corresponding sub-data; and calculate the mean and standard deviation of the L2 norm based on the L2 norm of all sub-data;

[0094] When the L2 norm of any sub-data is greater than or equal to the sum of the mean and standard deviation of the L2 norm, the privacy importance of the corresponding sub-data is determined to be high. The high-level sub-data is encrypted using a symmetric encryption algorithm to obtain the first encrypted data. When using the symmetric encryption algorithm, a small amount of error parameters are added to interfere to prevent reverse deduction.

[0095] When the L2 norm of any sub-data is greater than the difference between the mean and standard deviation of the L2 norm, and less than the sum of the mean and standard deviation of the L2 norm, the privacy importance of the corresponding sub-data is determined to be medium. The sub-data with the medium level is encrypted using an asymmetric encryption algorithm to obtain the second encrypted data;

[0096] When the L2 norm of any sub-data is less than or equal to the difference between the mean and standard deviation of the L2 norm, the privacy importance of the corresponding sub-data is determined to be low. The sub-data with a low level is encrypted using a homomorphic encryption algorithm to obtain third encrypted data.

[0097] S4: Use AI algorithms to perform threat detection on the different encrypted data of each medical institution, discard the threatening encrypted data, and re-execute steps S3-S4, during which the encryption algorithm is randomly switched for encryption; upload the non-threatening encrypted data to the central server for decryption and data aggregation to obtain the global model parameters.

[0098] In this embodiment, threat detection on encrypted data using an AI algorithm includes:

[0099] Data feature extraction: For the different encrypted data uploaded by medical institutions, we first need to extract its features. Since this data is the encrypted result of model updates, features can be extracted based on the data's encryption characteristics (such as parameters corresponding to the encryption algorithm type), data size, and transmission time interval.

[0100] Model training:

[0101] Prepare training data: Collect a large number of encrypted data samples known to contain threats to form a training dataset. This dataset includes encrypted data that is transmitted normally (no threats) and encrypted data that contains threats generated by simulated attacks or abnormal situations.

[0102] Select an appropriate AI model: Choose the LSTM model because it is more effective in detecting timing anomalies (such as sudden changes in transmission intervals and abnormal data bursts) during encrypted data transmission.

[0103] Training and Optimization: The extracted feature data is fed into the selected model. Model parameters are adjusted through algorithms such as backpropagation to minimize the error between the predicted result and the actual label (threat or no threat). During training, methods such as cross-validation can be used to prevent overfitting and improve the model's generalization capabilities.

[0104] Threat Detection: The encrypted data to be tested is characterized and fed into a trained model. Based on the learned patterns and rules, the model outputs a prediction of whether the encrypted data presents a threat. If it is determined to be a threat, the encrypted data is discarded, and the corresponding medical institution is required to re-execute the segmentation and encryption steps for the parameter update. If it is determined to be non-threatening, it is uploaded to the central server for further processing.

[0105] Dynamic update and optimization: As the medical data environment changes and new attack methods emerge, we continue to collect new encrypted data samples, regularly retrain the model, and update the model parameters to ensure the effectiveness and accuracy of the threat detection model.

[0106] Furthermore, the non-threatening encrypted data is uploaded to the central server for decryption and data aggregation, and the global model parameters are obtained, including:

[0107] After any threat-free encrypted data is uploaded to the central server, it is decrypted using a symmetric decryption algorithm, an asymmetric decryption algorithm, or a homomorphic decryption algorithm to obtain the corresponding decrypted data. When the decryption time exceeds the preset maximum decryption time, the corresponding encrypted data is marked as lost and threat detection is performed again.

[0108] Recombining the decrypted data belonging to the same medical institution into a second parameter update amount corresponding to the medical institution;

[0109] The mean of the second parameter update amounts of all medical institutions is calculated, and the global model parameters of this round of iteration are updated with the mean of the second parameter update amounts to obtain the global model parameters of the next round of iteration.

[0110] The mean of the second parameter update amount is used to update the global model parameters, making the update stable and representative, and avoiding the outliers of a single data sample from having too great an impact on the update.

[0111] In this embodiment, the preset maximum decryption time includes: maximum decryption time for low-level data / maximum decryption time for intermediate data / maximum decryption time for high-level data;

[0112] The maximum decryption time for low-level data is the maximum decryption time of the homomorphic decryption algorithm;

[0113] The maximum decryption time for intermediate data is the maximum decryption time of the asymmetric decryption algorithm;

[0114] The maximum decryption time for advanced data is the maximum decryption time of the symmetric decryption algorithm.

[0115] In this embodiment, the maximum decryption time for low-level data, the maximum decryption time for intermediate-level data, and the maximum decryption time for high-level data are obtained by training a large model.

[0116] In this embodiment, in order to provide better protection for data, key identification is also included before decryption:

[0117] Obtain the name, geographic location (latitude and longitude), environmental characteristics (temperature, humidity), device information (WiFi_MAC), and timestamp of each medical institution;

[0118] The name, geographic location, equipment information, and timestamp of any medical institution are synthesized (e.g., by simple concatenation) as a key seed. The key seed is unique and dynamic, and it is continuously updated as the timestamp changes.

[0119] The key seed and the corresponding environmental characteristics are passed through a key derivation function or a random number generator to generate a key. At the same time, the key needs to be trained adversarially. Through continuous adversarial training, the data can be protected with higher quality. Finally, before the key is used, the timeliness of the key is ensured through timestamp verification.

[0120] The key corresponding to each medical institution is uploaded to the central server together with the non-threatening encrypted data. The central server traces the source based on the key carried by each non-threatening encrypted data, queries the corresponding medical institution, and determines whether the corresponding medical institution is legal. The non-threatening encrypted data uploaded by the legal medical institution is decrypted and the non-threatening encrypted data uploaded by the illegal medical institution is discarded.

[0121] Through the proper use of keys, only authorized central servers can decrypt encrypted data, ensuring data security during transmission and storage, and preventing data from being accessed and interpreted by unauthorized third parties. Furthermore, the timestamp-based key generation is unique and dynamic. The central server traces the key to the medical institution and determines its legitimacy, preventing the infiltration of malicious data. This ensures that data participating in federated learning comes from trusted medical institutions, ensuring the security of the entire system and the reliability of the data.

[0122] S5: Each medical institution updates its local model parameters based on the global model parameters issued by the central server, and trains the local model based on the updated local model parameters.

[0123] In this embodiment, before the central server sends the global model parameters, it also includes: the central server evaluates the global model parameters. In addition to the accuracy Acc, the evaluation indicators also include the recall rate Recall, F1 value, etc. Taking the accuracy as an example, in the validation set Calculate on: , ( x , y ) represents the data samples and their labels in the validation set, Indicates that given the model parameters and data samples x In the case of z The likelihood of being the same is the greatest; Represents the model prediction data sample x Belong to category z probability; Indicates an indicator function that returns 1 if the argument is true, otherwise it returns 0. With label y When they are equal, the indicator function has a value of 1, otherwise it has a value of 0. If Acc, Recall, and F1 are all greater than or equal to the corresponding thresholds, the convergence condition is met and the global model parameters are distributed to each medical institution. Otherwise, the next round of iteration is continued.

[0124] The medical data privacy protection method based on federated learning and the BERT model provided in this embodiment has the following beneficial effects:

[0125] This method enables the sharing of medical data while protecting patient privacy information. At the same time, it reduces the communication overhead and computing power of each medical institution. Each medical institution updates its own local model parameters based on the issued global model parameters, and trains the local model of the corresponding medical institution based on the updated local model parameters. The trained local model can finally be applied to local data analysis and prediction tasks, further improving treatment efficiency.

[0126] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A medical data privacy protection method based on federated learning and BERT model, characterized in that: include: S1: Preprocess the local data of each medical institution to obtain the medical data set of each medical institution; S2: Each medical institution trains the BERT model issued by the central server based on its own medical dataset, calculates the corresponding model gradient, and perturbs each model gradient. Based on the perturbed model gradient, the parameter update amount for the corresponding medical institution is calculated, including: Add noise that obeys the Laplace distribution to the model gradient to obtain the perturbed model gradient; Multiply the perturbed model gradient by the learning rate to get the first product; The negative value of the first product is used as the parameter update amount of the corresponding medical institution; S3: Each medical institution randomly divides its own parameter update amount and classifies the divided data into levels according to the privacy importance. Different encryption algorithms are used to encrypt the data of each level to obtain the corresponding encrypted data; S4: Perform threat detection on each medical institution's encrypted data, discard any threatening encrypted data, and re-execute steps S3-S4; upload the non-threatening encrypted data to the central server for decryption and data aggregation to obtain global model parameters; S5: Each medical institution updates its local model parameters based on the global model parameters issued by the central server, and trains the local model based on the updated local model parameters.

2. The medical data privacy protection method based on federated learning and BERT model according to claim 1 is characterized in that Preprocessing of local data of any medical institution includes: The local data includes text data describing the patient's basic information and the disease the patient suffers from; Clean the local data of each medical institution separately, remove duplicate, missing, and non-compliant data, and fill in the data for the removed parts; Based on the disease types of patients in each local data, the professional classification of medical institutions is used to label the corresponding local data to obtain the disease department label corresponding to each local data; Use a specialized medical word segmenter to segment each filled local data, and convert each segmented local data into the vector format required by the BERT model. The transformed vectors are used as data samples, and the disease department labels corresponding to the local data are used as labels for the data samples to construct the medical data sets corresponding to the medical institutions.

3. The medical data privacy protection method based on federated learning and BERT model according to claim 2 is characterized in that Before the word segmentation process of the local data, dynamic data generalization is also performed on the padded local data: Generalize the specific age of patients in the filled local data into age ranges; The specific disease type of the patient in the filled local data is generalized into chronic disease / acute disease.

4. The medical data privacy protection method based on federated learning and BERT model according to claim 2 is characterized in that The BERT model sent by the central server includes: Obtain a large-scale professional corpus in the medical field, which includes a plurality of medical data samples and their corresponding first true labels; the content of the medical data sample corresponds to the content of the data sample, and the type of the first true label corresponds to the type of the label of the data sample; Input each medical data sample into the BERT model and output the corresponding first predicted label; Calculating a first cross entropy loss based on any first predicted label and the corresponding first true label, and adding a regularization term to the first cross entropy loss to obtain a loss function corresponding to the medical data sample; Sum the loss functions of all medical data samples to get the total loss, minimize the total loss, and get the pre-trained BERT model; Initialize the pre-trained BERT model and distribute the model parameters of the initialized BERT model to various medical institutions.

5. The medical data privacy protection method based on federated learning and BERT model according to claim 4 is characterized in that: Any medical institution trains the BERT model issued by the central server based on the corresponding medical dataset and calculates the corresponding model gradients including: Receive the model parameters of the initialized BERT model as the initial local model parameters of the corresponding medical institution; In the corresponding medical dataset, the label of each data sample is used as the second true label, and each data sample is input into the pre-trained BERT, which outputs the corresponding second predicted label; Calculate a second cross entropy loss based on any second predicted label and the corresponding second true label, and multiply the second cross entropy loss by the gradient operator to obtain the gradient of the second cross entropy loss of the corresponding data sample to the local model parameter; The gradients of the second cross entropy loss of all data samples with respect to the local model parameters are averaged to obtain the model gradients of the second cross entropy loss of the corresponding medical dataset with respect to the local model parameters.

6. The medical data privacy protection method based on federated learning and BERT model according to claim 1 is characterized in that S3 include: Any medical institution randomly divides its parameter update amount into several sub-data; Based on all elements in any sub-data, calculate the L2 norm of the corresponding sub-data; and calculate the mean and standard deviation of the L2 norm based on the L2 norm of all sub-data; When the L2 norm of any sub-data is greater than or equal to the sum of the mean and standard deviation of the L2 norm, the privacy importance of the corresponding sub-data is determined to be high, and the sub-data with high privacy importance is encrypted using a symmetric encryption algorithm to obtain first encrypted data; When the L2 norm of any sub-data is greater than the difference between the mean and standard deviation of the L2 norm, and less than the sum of the mean and standard deviation of the L2 norm, the privacy importance of the corresponding sub-data is determined to be medium. The sub-data with the medium level is encrypted using an asymmetric encryption algorithm to obtain the second encrypted data; When the L2 norm of any sub-data is less than or equal to the difference between the mean and standard deviation of the L2 norm, the privacy importance of the corresponding sub-data is determined to be low. The sub-data with a low level is encrypted using a homomorphic encryption algorithm to obtain third encrypted data.

7. The medical data privacy protection method based on federated learning and BERT model according to claim 6 is characterized in that Upload the threat-free encrypted data to the central server for decryption and data aggregation, and obtain the global model parameters including: After any threat-free encrypted data is uploaded to the central server, it is decrypted using a symmetric decryption algorithm, an asymmetric decryption algorithm, or a homomorphic decryption algorithm to obtain the corresponding decrypted data. When the decryption time exceeds the preset maximum decryption time, the corresponding encrypted data is marked as lost and threat detection is performed again. Recombining the decrypted data belonging to the same medical institution into a second parameter update amount corresponding to the medical institution; The mean of the second parameter update amounts of all medical institutions is calculated, and the global model parameters of this round of iteration are updated with the mean of the second parameter update amounts to obtain the global model parameters of the next round of iteration.

8. The medical data privacy protection method based on federated learning and BERT model according to claim 7 is characterized in that: The preset maximum decryption time includes: low-level data maximum decryption time / intermediate data maximum decryption time / high-level data maximum decryption time; The maximum decryption time for low-level data is the maximum decryption time of the homomorphic decryption algorithm; The maximum decryption time for intermediate data is the maximum decryption time of the asymmetric decryption algorithm; The maximum decryption time for advanced data is the maximum decryption time of the symmetric decryption algorithm.

9. The medical data privacy protection method based on federated learning and BERT model according to claim 7 is characterized in that Before decryption, key identification is also included: Obtain the name, geographic location, environmental characteristics, equipment information, and timestamp of each medical institution; Combine the name, geographic location, equipment information, and timestamp of any medical institution into a key seed; The key seed and the corresponding environmental characteristics are passed through a key derivation function or a random number generator to generate a key; The key corresponding to each medical institution is uploaded to the central server together with the non-threatening encrypted data. The central server traces the source based on the key carried by each non-threatening encrypted data, queries the corresponding medical institution, and determines whether the corresponding medical institution is legal. The non-threatening encrypted data uploaded by the legal medical institution is decrypted and the non-threatening encrypted data uploaded by the illegal medical institution is discarded.

Images