A teacher consensus aggregation learning method based on random response differential privacy technology

Abstract

The application discloses a teacher consensus aggregation learning method based on a random response differential privacy technology, and comprises the following steps: inputting a query into a teacher model to obtain an output label corresponding to the teacher model; wherein the teacher model is multiple, and each teacher model is different from each other; performing random disturbance on the output label to obtain a disturbed label; determining an estimated label corresponding to the query according to the disturbed label; wherein the estimated label is at least two; training a student model based on a data set composed of the query and the estimated label and a data set generated by a generative adversarial network to obtain a trained student model. Since the sensitive data in the sensitive data set cannot be obtained through the query and the estimated label when the student model is migrated, the problem of improving the privacy in the migration learning is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of risk prediction, and particularly relates to a teacher consensus aggregation learning method based on random response differential privacy technology. BACKGROUND

[0002] Data-driven learning models are constructively combined with the development of computational and analytical methods, fully exploiting the full power of massive medical and health data, thus bringing new insights into healthcare, clinical decision support and disease risk prediction. Generally, these algorithms rely heavily on a large number of labeled medical data to build classifiers or prediction models for effective secondary use, so the quality and quantity of training data obviously have a great influence on the training results. However, due to the low incidence of some diseases or the long medical observation period, qualified data sets of these diseases are not always available, and there may be some related chronic diseases with large amounts of data that can provide useful information for target inference.

[0003] Transfer learning is a technique that can achieve the above-mentioned goal, which explores the possibility of using prior knowledge learned from one field for another field. According to the content to be transferred, these methods can be roughly divided into several categories, including instance transfer, feature transfer and parameter transfer. Inevitably, in the transfer process, the target domain needs sample data or model information of the source domain to improve the model, but it is not always allowed to directly transfer such sensitive information, especially for the case where the source data to be transferred is owned by different external medical institutions. Recent work has shown that machine learning algorithms are vulnerable to malicious privacy attacks, and traditional data model training is not a one-way process, and model inversion and membership inference can be used as black-box attacks (attackers have only query access to the model) or white-box attacks (attackers can have full access to the structure and parameters of the model). The generated sequence learning model may inadvertently remember the training data, which makes it possible for an adversary to extract sensitive information of a deep learning participant. Therefore, in the prior art, there is a privacy problem when any source domain information is shared with an external medical institution that wants to use it to improve its local target learning performance.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The technical problem solved by the present application is to provide a teacher consensus aggregation learning method based on random response differential privacy technology to solve the problem of low privacy in transfer learning in the prior art in view of the above-mentioned defects of the prior art.

[0006] The technical solution adopted by the present application to solve the technical problem is as follows:

[0007] A teacher consensus aggregation learning method based on random response differential privacy technology, comprising the following steps:

[0008] inputting a query into a teacher model to obtain an output label corresponding to the teacher model; wherein, there are multiple teacher models, and each teacher model is different from each other;

[0009] randomly perturbing the output label to obtain a perturbed label;

[0010] determining an estimated label corresponding to the query according to the perturbed label; wherein, there are at least two estimated labels;

[0011] training a student model based on a data set composed of the query and the estimated label and a data set generated by a generative adversarial network to obtain a trained student model.

[0012] The teacher consensus aggregation learning method based on random response differential privacy technology, wherein each teacher model is trained by using a sensitive data subset corresponding to the teacher model, and the sensitive data subset is divided according to a sensitive data set.

[0013] The teacher consensus aggregation learning method based on random response differential privacy technology, wherein the determination of the estimated label corresponding to the query according to the perturbed label comprises:

[0014] determining a label count of the perturbed label according to the perturbed label;

[0015] determining an unbiased estimator of each label count according to the label count;

[0016] determining the estimated label corresponding to the query according to the unbiased estimator.

[0017] The teacher consensus aggregation learning method based on random response differential privacy technology, wherein the unbiased estimator of each label count is:

[0018]

[0019] wherein, j represents a label number, x represents a query, represents an unbiased estimator of a label count of the jth label obtained by predicting and perturbing the query x by n teacher models, p -1 represents an inverse of a design matrix, represents a proportion of the perturbed label in all labels, and n represents a number of teacher models.

[0020] The teacher consensus aggregation learning method based on random response differential privacy technology, wherein the design matrix is:

[0021] p={p uv}

[0022]

[0023] u,v∈(1,2,...,t)

[0024] Wherein, p uv Indicates the element of the design matrix, u indicates the disturbance label, v indicates the output label, e indicates the natural constant, t indicates the label, t≥2, and ε indicates the privacy budget.

[0025] The teacher consensus aggregation learning method based on the random response differential privacy technology, wherein the estimated label is:

[0026]

[0027] Wherein, Indicates the estimated label, and argargmax(·) indicates the variable value when the target function takes the maximum value.

[0028] A method for establishing a stroke risk prediction model, comprising the steps of:

[0029] In the source domain, the teacher consensus aggregation learning method based on the random response differential privacy technology is used to obtain a trained student model.

[0030] The trained student model is migrated to the target domain for training to obtain a stroke risk prediction model.

[0031] The method for establishing a stroke risk prediction model, wherein the trained student model is migrated to the target domain for training to obtain a stroke risk prediction model, comprising:

[0032] For each feature in the feature space, according to the source domain correlation coefficient vector and the target domain correlation coefficient vector, the similarity corresponding to the feature is determined;

[0033] According to the source domain correlation coefficient vector and the similarity, the similarity weight corresponding to the feature is determined;

[0034] According to the respective similarity weights corresponding to each feature, the target feature is determined.

[0035] The trained student model is used as a pre-training model, and the pre-training model is trained based on the target feature to obtain a stroke risk prediction model.

[0036] The method for establishing a stroke risk prediction model, wherein the similarity is:

[0037]

[0038] ω Ti , ω Si ∈[-1, 1]

[0039] wherein, denotes the similarity corresponding to the i th feature, ω Ti denotes the i th element of the correlation coefficient vector in the target domain, ω Si denotes the i th element of the correlation coefficient vector in the source domain.

[0040] The brain stroke risk prediction model establishment method, wherein the similarity weight is:

[0041]

[0042] wherein, Δ i denotes the similarity weight, and alpha and beta are weight coefficients.

[0043] Beneficial effects: since the sensitive data in the sensitive data set cannot be obtained by querying and estimating the label when the student model is migrated, the privacy problem in the migration learning is improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the first flow chart of the teacher consensus aggregation learning method based on the random response differential privacy technology in the application.

[0045] Figure 2 is the first flow chart of the brain stroke risk prediction model establishment method in the application.

[0046] Figure 3 is the accuracy rate graph of predicting hypertension under different perturbation probabilities and teacher numbers in the application.

[0047] Figure 4 is the label reversal frequency graph under different perturbation probabilities and teacher numbers in the application.

[0048] Figure 5 is the test precision comparison of RRML and PATE-G in the application.

[0049] Figure 6 is the label reversal number comparison of RRML and PATE-G in the application.

[0050] Figure 7 is the accuracy comparison of RRML and the migration learning based on PATE-G in the application.

[0051] Figure 8 is the second flow chart of the teacher consensus aggregation learning method based on the random response differential privacy technology in the application.

[0052] Figure 9 is a second flow chart of the method for establishing a stroke risk prediction model in the present application. DETAILED DESCRIPTION

[0053] In order to make the objects, technical solutions and advantages of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0054] Differential privacy is a widely used strict mathematical concept that can measure the privacy loss budget for data release mechanisms. Based on differential privacy, a privacy-preserving machine learning method is designed, and Papernot et al. proposed a universally applicable differential privacy machine learning algorithm, i.e. PATE algorithm, which can ensure that the training data is safe from model inversion and member inference attacks. However, PATE directly adds Laplace noise to the voting results, which inevitably leads to lower prediction accuracy. Although better privacy-utility trade-offs are provided under the PATE infrastructure by applying new noise aggregation mechanisms or new training methods, these algorithms all assume the existence of a trusted model trainer (such as a researcher), which is not always the case. In fact, data owners and model trainers usually belong to different institutions, i.e. training work is done by a third party in a non-local manner, because medical institutions usually use limited computing and analysis resources for configuration and only have the ability to handle a small amount of data. That is, there are still privacy issues.

[0055] Please also refer to Figures 1-9 The present application provides some embodiments of a teacher consensus aggregation learning method based on random response differential privacy technology. The teacher consensus aggregation learning method based on random response differential privacy technology in the present embodiment is a local differential private machine learning algorithm (RRML) based on a random response mechanism.

[0056] As shown in Figure 1 and Figure 8 , the teacher consensus aggregation learning method based on random response differential privacy technology in the present embodiment includes the following steps:

[0057] Step S100, input a query into a teacher model to obtain an output label corresponding to the teacher model; wherein the teacher model is multiple, and each teacher model is different from each other.

[0058] Specifically, there are multiple teacher models, that is, there are at least 2 teacher models, and each teacher model is different from each other, so the output labels corresponding to each teacher model are also not exactly the same.

[0059] For example, in this embodiment, there are n teacher models, which are Teacher 1, Teacher 2, …, Teacher i, …, Teacher n, where i ∈ (1, 2, …, n). The query x is input into each teacher model to obtain the output label corresponding to each teacher model, and then n output labels can be obtained, which are f 1 (x), f 2 (x), …, f i (x), …, f n (x). The output label of the i th teacher model is f i (x). i (x), the output label of the i th teacher model is f i (x). n (x), the output label of the i th teacher model is f i (x). i (x), the output label of the i th teacher model is f i (x). i (x) can be one of the t labels, for example, f i (x) = j. i (x) can be one of the t labels, for example, f i (x) = j.

[0060] Specifically, each teacher model is trained using the sensitive data subset corresponding to the teacher model, which is divided from the sensitive data set. The sensitive data set refers to a data set that involves user information and needs to be protected. Taking a hospital as an example, the sensitive data in the sensitive data set includes user information such as age, gender, occupation, and illness, etc. Specifically, the type of disease can be hypertension, stroke, etc., and the sensitive data also includes data such as blood pressure value of hypertension.

[0061] Because the sensitive data subsets used to train each teacher model are different, the teacher models obtained are also different. When training the teacher models, the sensitive data set is first divided into n sensitive data subsets, and then each teacher model is trained based on each sensitive data subset.

[0062] For example, the sensitive data set (Sensitive Data) is divided into n parts, and each sensitive data subset is Data 1, Data 2, …, Data i, …, Data n. Each teacher model is trained based on each sensitive data subset. Because each sensitive data subset is not completely the same, the parameters of each teacher model obtained by training are also different. It should be noted that the number of sensitive data subsets and the number of teacher models are the same.

[0063] Step S200, randomly perturbing the output label to obtain a disturbed label.

[0064] Specifically, the output label is randomly perturbed to obtain a disturbed label, which can be the same as the output label before disturbance, or can be different from the output label before disturbance.

[0065] For example, the output label f i (x) is randomly disturbed to obtain a disturbed label f′ i (x) i (x) has a probability P equal to f i (x) becomes other labels with a probability 1-P, that is, f′ i (x) = f i (x) has a probability p; f′ i (x) ≠ f i (x) has a probability 1-P.

[0066] For example, f i (x) = j, that is, the output label f i (x) output by the i-th teacher model is label j. The probability of the disturbed label f′ i (x) = j is P, that is, the probability that the disturbed label after disturbance of the output label output by the i-th teacher model is the same as the output label is P. The probability of the disturbed label f′ i (x) ≠ j, that is, the probability that the disturbed label after disturbance of the output label output by the i-th teacher model is not the same as the output label is 1-P.

[0067] Step S300, determining an estimated label corresponding to the query according to the disturbed label; wherein the estimated label has at least two.

[0068] Specifically, the estimated label corresponding to the query is determined according to the disturbed label, and the estimated label has at least two, that is, the estimated label obtained according to the query is not unique, and even if the query and the estimated label are obtained, the sensitive data in the sensitive data subset cannot be obtained, thereby ensuring the privacy of the data.

[0069] Specifically, the estimated label is obtained by counting and correcting the disturbed label. Specifically, step S300, determining an estimated label corresponding to the query according to the disturbed label, comprises:

[0070] Step S310, determining a label count of the disturbed label according to the disturbed label.

[0071] Specifically, the label count of the output label is not the same as the label count of the disturbed label, and in this embodiment, the label count refers to the label count of the disturbed label unless otherwise specified.

[0072] For example, there are t labels, which are 1, 2,..., v,..., u,..., t. The v-th label count in the output label is:

[0073] nv (x)=|{i:i∈[n],f i (x) = v}|, v∈[t]

[0074] The count of the u-th label in the scrambling labels is:

[0075] n′ u (x)=|{i:i∈[n],f′ i (x) = u}|, u∈[t]

[0076] Since the output label is not exactly the same as the perturbed label after random perturbation, for the same label j, the label count n of the output label is... j (x) and the tag count n′ of the scrambling tags j (x) are not the same.

[0077] Step S320: Determine the unbiased estimate of each tag count based on the tag count.

[0078] Specifically, based on the label counts of the scrambled labels, unbiased estimators for each label count are determined. In other words, unbiased estimators for each label count are obtained by correcting the label counts. Then, the estimated label corresponding to the query is determined using the unbiased estimators.

[0079] The output label f ​​of the i-th teacher model i (x) represents the v-th label v and the output label f ​​of the i-th teacher model. i (x) The perturbation label f′ obtained after perturbation i (x), disrupting the label f′ i (x) represents the u-th label u. uv The output label f ​​before the disturbance i (x) = v is perturbed to obtain the perturbation label f′. i (x)=u, where u, v=(1,...,t), This indicates the proportion of the disruptive tag among all tags, for example, This represents the proportion of the u-th tag u in all tags in the scrambling tags, that is, the proportion of the u-th tag u in the scrambling tags to the total number of tags n. The scrambling tag f′ i The number of labels u in (x) is n′ u (x), then By counting the number of each label in the scrambled labels and the total number of all labels n, we can obtain the result.

[0080] Define λ = (λ1, ..., λ2) t ) T , π=(π1,...,πt T , and has:

[0081] λ = pπ

[0082] where p = {p uv}, that is λ represents the proportion of the disturbance label in all labels, π represents the unbiased estimate of the label count of the disturbance label, and p represents the design matrix.

[0083] In the data collection scenario, compared with the standard Laplace mechanism, the random response technique can provide better statistical utility, especially when the privacy budget is very limited. In order to maximize the sum of the diagonal elements, the design matrix p = {p uv} of the random response should adopt the following format:

[0084]

[0085] u, v ∈ (1, 2,..., t)

[0086] where p uv represents the element of the design matrix, u represents the disturbance label, v represents the output label, e represents the natural constant, t represents the label, t ≥ 2, ε represents the privacy budget, and the value range of the privacy budget is usually 0-10. If the privacy budget exceeds 10, it is considered that the privacy is not guaranteed, and the privacy budget can be set according to the needs. The privacy budgets corresponding to different queries can be the same or different. After the privacy budget is determined, the design matrix can be determined.

[0087] Therefore, there is π = p -1 λ, and it should be noted that there is no sample number here. For a simple random sample replaced by n, is the sample proportion corresponding to λ, for example represents the proportion of the disturbance label in all labels, and of course represents the sample proportion of the disturbance label u in all disturbance labels. Then assume that the design matrix p is non-singular, represents the unbiased estimate of the label count, since according to the above derivation, there is π = p -1 λ, then there is then the unbiased estimate corresponding to the label count of each label is:

[0088]

[0089] Specifically, each label corresponds to an The corresponding of each label is different, is the unbiased estimate of n j ​The unbiased estimator of (x) is obtained, that is, although n j The difference between (x) and n j The difference between (x) and n The difference between (x) and n j The difference between (x) and n

[0090] Since the output label needs to be disturbed to ensure privacy, the disturbed label obtained after disturbance is quite different from the output label, so the disturbed label needs to be corrected to obtain the estimated label, so that the difference between the estimated label and the output label is small, and the estimated label and the query can be used as a data set.

[0091] Step S330, determining the estimated label according to the unbiased estimator.

[0092] Specifically, the estimated label is determined according to the unbiased estimator, and the label when the unbiased estimator takes the maximum value is taken as the estimated label. Specifically, the estimated label is:

[0093]

[0094] Wherein, represents the estimated label, and argargmax(·) represents the variable value when the objective function takes the maximum value. Specifically, the label corresponding to the maximum value of is taken as the estimated label, that is, looking at the unbiased estimator of the label count corresponding to each label, the maximum value of the unbiased estimator of the label count, and the label is the estimated label. Since the disturbed and corrected The difference between (x) and n j The difference between (x) and n

[0095] Step S400, training the student model based on the data set composed of the query and the estimated label and the data set generated by the generative adversarial network to obtain the trained student model.

[0096] Specifically, the student model is trained by the data set composed of the query and the estimated label and the data set generated by the generative adversarial network (GAN). When the student model is migrated, the sensitive data in the sensitive data set cannot be obtained through the query and the estimated label, so as to improve the privacy in the migration learning.

[0097] To establish strong privacy protection, the number of queries of the teacher model should be limited, because for each query, we add a new perturbation, which generates a new privacy budget. Therefore, the generative adversarial networks (GANs) are applied to semi-supervised learning. Like all semi-supervised learning methods, the RRML framework that incorporates semi-supervised mechanism assumes that the student model has access to some unlabeled data. This assumption does not greatly limit the applicability of the method: even when learning sensitive data, there are often non-overlapping, unlabeled data sets from which semi-supervised methods can extract prior distributions. For example, there are public data sets for text and images as well as medical data.

[0098] Specifically, since the student model is trained based on the data set composed of the query and the estimated label and the data set generated by the generative adversarial network, the accuracy of the transfer learning can be improved, and the accuracy of the method in the embodiment is higher than that of the generally applicable differential privacy machine learning method.

[0099] Privacy analysis of RRML.

[0100] Before performing the privacy analysis, two concepts, differential privacy and moment accountant, should be understood.

[0101] ①Differential privacy

[0102] For a random algorithm A: D→R, given any sibling databases d, d'∈D (|dΔd'|≤1) and any output subset S If it satisfies differential privacy ((ε, δ)-DP), then:

[0103] Pr[A(d)∈S]≤e ε Pr[A(d')∈S]+δ

[0104] Where ε and δ are two non-negative real values, Pr[·] represents probability, D represents input data, and R represents output data.

[0105] ②Moment accountant

[0106] For an output o∈R, the privacy loss on o is defined as:

[0107]

[0108] The privacy loss random variable C(A, aux, d, d') is defined as c(A(d); A, aux, d, d'), where aux represents additional input. The privacy loss is calculated using the moment accountant. For any constant λ, for example, λ=8 can be used. The definition of the moment accountant is as follows:

[0109]

[0110] where is the generating function of the privacy loss random variable, represents the accounting at time t has the following properties:

[0111] T HEOREM 1 (composability). Assume that a randomized algorithm A consists of a sequence of sub-algorithms A1,... A k where then for any output sequence o1,... o k-1 and any λ have:

[0112]

[0113] T HEOREM 2 (tail bound). For any ε > 0, the algorithm A satisfies differential privacy then has:

[0114]

[0115] the probability of correctly reconstructing the value of the query x as v from the perturbed data.

[0116] The matrix p = {p uv} is called the design matrix, where the sum of each column in p is 1:

[0117]

[0118] To maximize the sum of the diagonal elements, the design matrix p = {p uv} of the randomized response should take the following form:

[0119]

[0120] Let denote the probability of correctly reconstructing the value of the query x as v from the perturbed data, P(y i = u | x i = v) = P uv denotes the true output value of the i-th teacher model x i is v, and after perturbation y i becomes u, where u, v ∈ (1, 2,..., t).

[0121]

[0122] According to Bayes' theorem, we have:

[0123]

[0124] Introduce the accounting at time t to combine the privacy of multiple queries.

[0125] To obtain the overall privacy loss of RRML, we need to calculate and f(x) is different, i.e. the probability that the corrected voting label obtained after the disturbance is different from the actual voting result. Assuming that this probability is q, obviously The total privacy budget of multiple queries is tightened by using time accounting, and the following theorem holds:

[0126] T HEOREM 3 (time accounting). A is a random algorithm satisfying ε-DP, ε, l ≥ 0, for sibling databases d, d' have:

[0127]

[0128] Finally, the total privacy budget ε total Limited by theorem 2.

[0129] The application also provides a preferred embodiment of a method for establishing a stroke risk prediction model.

[0130] Transfer learning extracts useful knowledge from auxiliary data in related fields or tasks to improve the performance of target tasks. Among them, the domain (domain) is defined as: Where Refers to the feature space, P(X) represents the marginal probability distribution. The task (task) includes a label space And the conditional probability distribution P(Y|X) usually obtained from the training data samples {x i , y i}, where x S ∈X, Given a target domain and task Transfer learning aims to use the prior knowledge P(Y S |X T ) retained in the related source domain To facilitate learning in the conditional probability distribution P(Y T |X S ) in It is mentioned above that machine learning models are vulnerable to attacks in membership inference attacks and model inversion attacks, and transfer learning must involve model sharing from source to target, which will cause many privacy problems. Therefore, using the RRML mechanism to construct a privacy protection transfer learning infrastructure (RRMTL) can solve the privacy problem in knowledge transfer. And application in improving the performance of stroke risk prediction.

[0131] The basic architecture of RRMTL is as follows: Figure 2As shown, the student model is first trained in the source domain (hypertension data) using the RRML structure. Similar to PATE-G, a generative adversarial network (GAN) is also adopted to generate artificial hypertension samples to optimize performance, because the query to the teacher model is reduced, and the privacy loss is also reduced. Then the student model weights are migrated as a pre-trained model to the risk prediction task in the target domain (stroke data) to improve accuracy. The RRMTL basic structure can obtain valuable knowledge in the source domain without infringing on the private data in the source domain, which is mainly achieved in the following two aspects:

[0132] (1) Since RRML provides strict differential privacy protection, even if the entire student model can be accessed, the attacker cannot obtain any sensitive information of a single training sample in the source domain;

[0133] (2) The local differential privacy protection function of RRMTL can ensure that the attacker cannot determine whether a certain training point exists in the source domain training set.

[0134] As shown in Figure 2 and Figure 9 , the method for establishing a stroke risk prediction model according to the embodiments of the present application comprises the following steps:

[0135] Step A100, in the source domain, a teacher consensus aggregation learning method based on random response differential privacy technology according to any one of the above embodiments is used to obtain a trained student model.

[0136] Specifically, the source domain is determined according to the stroke risk prediction model, and is usually selected according to the type of disease. For example, hypertension, Alzheimer's disease, which is similar to stroke. It should be noted that the source domain usually has a sufficient number of high-dimensional data. In this embodiment, hypertension data is used as the source domain data.

[0137] Step A200, migrating the trained student model to the target domain for training to obtain a stroke risk prediction model.

[0138] Specifically, as described above, the source domain usually has a sufficient number of high-dimensional data. However, there may be some irrelevant feature data in these data, and such features may increase the distance between domains and cause the transfer accuracy to decrease. There have been many works on domain adaptability, for example, using domain similarity and domain complexity to select an instance subset, and for example, using Bayesian optimization to select data. In this embodiment, the similarity of feature weights is used to find a set of optimized features to avoid negative transfer and thus obtain better transfer performance. The feature optimization algorithm (FOA) is used to calculate the weight of each feature in the source domain and the target domain, and then the features with greater similarity are selected for transfer.

[0139] Step A200, migrating the trained student model to the target domain for training to obtain a stroke risk prediction model, comprising:

[0140] Step A210, for each feature in the feature space, determining the similarity of the feature corresponding to the source domain correlation coefficient vector and the target domain correlation coefficient vector.

[0141] Specifically, assuming the feature space X S = X T = {x1,..., x l}, where |X S | = |X T | = 1. The correlation coefficient matrices ω S and ω T of the source domain and the target domain are calculated respectively. The vector is used to calculate the similarity of the feature x i between the source domain task and the target task.

[0142] The similarity is:

[0143]

[0144] ω Ti , ω Si ∈ [-1, 1]

[0145] wherein, represents the similarity corresponding to the i-th feature, ω Ti represents the i-th element of the correlation coefficient vector in the target domain, and ω Si represents the i-th element of the correlation coefficient vector in the source domain.

[0146] Step A220, determining the similarity weight corresponding to the feature according to the source domain correlation coefficient vector and the similarity.

[0147] Specifically, for each feature, the similarity weight corresponding to the feature is determined according to the source domain correlation coefficient vector and the similarity corresponding to the feature.

[0148] The similarity weight is:

[0149]

[0150] wherein, Δ i represents the similarity weight, and α, β are weight coefficients, which depend on the training data itself.

[0151] Step A230, determining the target feature according to the similarity weight corresponding to each feature respectively.

[0152] Specifically, Δi The larger the value, the higher the value of the feature. and The stronger the correlation between features, the better. The selected features will be used to train the source domain model and the target domain model. A similarity weight threshold can be set; when the similarity weight of a feature is greater than or equal to the threshold, that feature will be used as the target feature. When the similarity weight of a feature is less than the threshold, that feature will not be used as the target feature.

[0153] Step A240: Use the trained student model as a pre-trained model, and train the pre-trained model based on the target features to obtain a stroke risk prediction model.

[0154] Specifically, the trained student model is used as the pre-trained model. That is, the weights obtained by the trained student model in the source domain are retained, and the pre-trained model is trained based on the target features to obtain the stroke risk prediction model.

[0155] To validate the proposed RRML scheme and the stroke risk prediction method based on RRMTL, experiments were conducted in a real-world scenario. The data used in the experiments were collected from the EHR databases of a hospital and a medical institution located in the same city. The source domain data consisted of 20,000 hypertension cases recorded by the hospital from 2012 to 2015. For the medical institution, given the 2,000 stroke events recorded from 2012 to 2015, the objective was to predict the risk of stroke events from 2016 to 2018. The experiments mainly consisted of the following four parts.

[0156] (1) The accuracy of the RRML algorithm in testing hypertension data. Figure 3 The test accuracy of the proposed RRML scheme is shown under different teacher models and perturbation probabilities. The three lines represent the overall accuracy trends when n=5, 50, and 100, respectively. As can be seen from the figure, the accuracy of hypertension prediction gradually decreases with increasing perturbation probability. Generally, the model achieves better results when the number of teacher models is set to n=50 compared to setting n to a value that is too small (n=5) or too large (n=100). Figure 4 The number of unsuccessful (or reversed) votes in the teacher voting results is given. A higher probability of interference (p = 0.3) leads to more label inversions, which also explains... Figure 3 The accuracy trend is shown. When n is set to a smaller number, the failure rate is more sensitive to the probability of interference. However, too large a value for n can also lead to a decrease in overall test accuracy, because in this case, each teacher model will be trained on a smaller subset of data.

[0157] (2) Performance comparison between RRML and PATE-G. The PATE-G scheme utilizes Laplacian noise to interfere with the voting results of the teacher model, while RRML adopts the random response technique as the interference mechanism. To evaluate the effectiveness of the scheme, the test accuracy Figure 5 ) and the number of label reversals Figure 6 ) of the proposed RRML and PATE-G schemes are compared in this experiment. As shown in Figure 5 , RRML is significantly better than PATE-G in terms of test accuracy when the privacy budget ∈ total ≤ 1.5. The average test accuracy of RRML (i.e., Pro in Figure 5 ) is 87%, while the average accuracy of PATE-G (i.e., lap in Figure 5 ) is less than 63%. The number of label reversals shown in Figure 6 indicates that the label failure rate of random response will be much smaller compared with the Laplacian noise mechanism. In short, the proposed RRML (i.e., Pro in Figure 6 ) is significantly better than the PATE-G (i.e., lap in Figure 6 ) scheme in providing a high level of data utility, especially when the privacy budget is small (e.g., ≤ 1.5).

[0158] (3) Effect of FOA. Table 1 shows the performance of the stroke risk prediction model based on RRMTL. Without FOA, the accuracy of the stroke risk prediction model directly transferred from the hypertension domain will even decrease from 73.2% to 72.3%, which is a negative transfer. By applying FOA to transfer learning, the model has achieved significant performance improvement in terms of precision (73.2% to 80.5%), recall (69.5% to 75%), and recall (80.4% to 84.2).

[0159] Table 1 Application of FOA in transfer learning

[0160]

[0161] F value is the abbreviation of F-measure, which refers to the weighted average of precision and recall, and AUC is the abbreviation of Area Under roc Curve, which refers to the area value surrounded by ROC (Receiver Operating Characteristic Curve) and coordinate axis.

[0162] (4) Performance of the stroke risk prediction model based on RRMTL. To evaluate the performance of the proposed stroke risk prediction model based on RRMTL, extensive experiments have been conducted. The source domain data (i.e., 20,000 hypertension samples) are divided into multiple independent blocks to facilitate the proposed RRML scheme to obtain multiple teacher models. For the training of the student model, 800 queries are used to obtain the corresponding privacy labels. These data are combined with the GAN-generated artificial samples to train the student model in a semi-supervised manner. The weights of the student model for predicting hypertension are then migrated to the target domain. As shown in Table 2, under the privacy protection architecture of RRTML, the performance evaluation parameters of the stroke risk prediction model are all improved to varying degrees. In this experiment, the total privacy budget of all RRMTL-based stroke risk prediction implementations is set to ε total ≤ 1.5. To better evaluate the proposed RRMTL, a stroke risk prediction model based on PATE-G for transfer learning is also implemented for comparison. The main setting difference between these two groups of experiments is the privacy protection machine learning algorithm used for hypertension model training, i.e., RRML or PATE-G. As shown in Figure 7 , for the case where the number of teacher models is equal to 5, 50, and 100, the stroke risk prediction model based on RRMTL (i.e., pro in Figure 7 ) has higher accuracy than the implementation based on PATE-G (i.e., lap in Figure 7 ). Compared with the clean transmission implementation without privacy protection (i.e., clean in Figure 7 ), only a slight decrease in precision can be observed. Therefore, the proposed stroke risk prediction model based on RRTML provides a better data privacy-utility balance.

[0163] Table 2 Performance of the stroke risk prediction model based on RRMTL (with FOA)

[0164]

[0165] The present application has the following three advantages:

[0166] 1. A classification method based on random perturbation differential privacy aggregated teacher ensemble (RRML) is proposed. Compared with the standard Laplace noise mechanism that directly uses noise statistics to train the student model, random response uses maximum likelihood estimation to approximate the true aggregated teacher label. It is proved both formally and empirically that RRML will obtain better utility when applied to private data classification problems, especially when the privacy budget is small.

[0167] 2. A local differential private data fusion and knowledge transfer infrastructure, RRTML, is proposed. Compared with other output perturbation mechanisms (e.g., PATE and its variants), it guarantees local differential privacy without assuming that the model trainers are trustworthy. Therefore, it is more suitable for adoption in a highly privacy-conscious federated learning scheme.

[0168] 3. A stroke risk prediction model using transfer learning is implemented under the RRTML infrastructure. A hypertension prediction model is first trained and then transferred to improve the performance of the stroke prediction model. A real dataset containing 2426 stroke events recorded in three local hospitals between 2012-2017 is used to test it. The experimental results show that the proposed scheme improves the stroke prediction accuracy from 73% to 81% with a privacy guarantee of ε≤1.5.

[0169] It should be understood that the application of the present application is not limited to the above examples, and can be improved or changed according to the above description for those of ordinary skill in the art, and all such improvements and changes shall fall within the scope of protection of the claims appended to the present application.

Claims

1. A teacher consensus aggregation learning method based on a random response differential privacy technology, characterized in that, The method comprises the steps of: inputting query data into a teacher model to obtain an output label corresponding to the teacher model; wherein the teacher model is multiple, and each teacher model is different from each other; randomly perturbing the output label to obtain a perturbed label; determining an estimated label corresponding to the query data according to the perturbed label; wherein the estimated label is at least two; training a student model based on a data set composed of multiple query data and respective corresponding estimated labels and a data set generated by a generative adversarial network to obtain a trained student model; each teacher model is trained by using a sensitive data subset corresponding to the teacher model, and the sensitive data subset is formed according to a sensitive data set; determining the estimated label corresponding to the query data according to the perturbed label comprises: determining a label count of the perturbed label according to the perturbed label; determining an unbiased estimator of each label count according to the label count; determining the estimated label corresponding to the query data according to the unbiased estimator; the unbiased estimator of each label count is: wherein, denotes a label index, denotes a query data, denotes a query data an unbiased estimator of the label count of the j label after n teacher model predictions and perturbations, denotes the inverse of the design matrix, denotes the proportion of the perturbed label among all labels, denotes the number of teacher models; the design matrix is: wherein, denotes an element of the design matrix, denotes a perturbation label, denotes an output label, denotes a natural constant, denotes a label, , denotes a privacy budget.

2. The teacher consensus aggregation learning method based on random response differential privacy technology according to claim 1, characterized in that, the estimated label is: wherein, denotes the estimated label, denotes the variable value at which the objective function takes its maximum. 3.A method for establishing a stroke risk prediction model, characterized in that, The method comprises the steps of: using the teacher consensus aggregation learning method based on the random response differential privacy technology in any one of claims 1-2 in the source domain to obtain a trained student model; migrating the trained student model to the target domain for training to obtain a stroke risk prediction model; wherein the target domain is stroke data, and the source domain is data of chronic diseases related to stroke. 4.The method of claim 3, wherein, The method of migrating the trained student model to the target domain for training to obtain a stroke risk prediction model comprises: for each feature in the feature space, determining the similarity of the feature according to the source domain correlation coefficient vector and the target domain correlation coefficient vector; determining the similarity weight of the feature according to the source domain correlation coefficient vector and the similarity; determining the target feature according to the respective similarity weight of each feature; using the trained student model as a pre-training model, training the pre-training model based on the target feature to obtain a stroke risk prediction model. 5.The method of claim 4, wherein, The similarity is: in, Indicates the first i The similarity corresponding to each feature The first element of the correlation coefficient vector in the target domain is... i One element, The first element representing the correlation coefficient vector in the source domain is... i Each element. 6.The method of claim 5, wherein, the similarity weight is: wherein denotes a similarity weight, are weight coefficients.

Images