A Naive Bayes Classification Method under Local Differential Privacy Model

Abstract

The present invention discloses a naive Bayes classification method under a local differential privacy model, which belongs to the field of information security. First, a user randomly samples a feature from multiple features, and the value of the feature and the value of a class label form a two-dimensional data. Secondly, according to the value of the privacy budget provided by the server, the user locally perturbs his or her two-dimensional data. Then, the server collects the perturbation data of all users, and estimates the frequency of the class label and the joint distribution between the feature and the class label. Finally, a naive Bayes classifier is used to classify and predict the test data. The present invention takes into account the correlation between features and class labels, designs a perturbation mechanism and a corresponding estimation mechanism, and theoretically analyzes a better way to divide the privacy budget. This method not only ensures the privacy security of users, but also ensures the accuracy of the statistical results of the data collector.

Description

Technical Field

[0001] The present invention belongs to the field of information security, and in particular relates to a naive Bayes classification method under a local differential privacy model. Background Art

[0002] With the continuous development of big data, data integration and mining are becoming more and more convenient. At the same time, user information security issues have become the focus of attention. The information, browsing history, life trajectory, etc. left by users on the APP will be stored, calculated, and collected into big data by the Internet. At present, governments, enterprises, and related researchers are using this data to obtain effective knowledge and information.

[0003] In order to better protect users' personal information, differential privacy [C.Dwork, "Differential privacy," in ICALP.Springer, 2006, pp.1–12.] has become a standard for publishing private data. When an attacker has strong background knowledge, he can use the difference between the first n-1 and the first n records to obtain the private information of the nth person. At this time, the concept of differential privacy is introduced. Centralized differential privacy is to transmit user information to the data collector (trusted third party). However, trusted third parties are rare in real life, so centralized differential privacy has certain limitations. On this basis, local differential privacy [Q.Ye and H.Hu, "Local differentialprivacy: Tools, challenges, and opportunities," in WISE.Springer, 2020, pp.13–23.] (LDP) was developed. LDP is based on an untrusted third party. Users protect data locally and send the perturbed data to the data collector. The data collector extracts relevant information based on the perturbed data received. In recent years, LDP has attracted widespread attention in academia and has also been used in industry. For example, Google uses RAPPOR technology to collect user behavior statistics from the Chrome browser, and Apple uses local differential privacy technology in Emoji, QuickType input suggestions, and search tips.

[0004] At present, data mining and machine learning have become commonly used research tools in data analysis. They are widely used in real life for spam filtering and prediction of various problems. The naive Bayes method is a classification method based on Bayes' theorem and the assumption of conditional independence of features. Its main idea is to solve the conditional probability of each category for the given items to be classified, and take the largest one as the classification result. When collecting data, users are often reluctant to report their private data, so it is impossible to accurately count useful information, which limits the application of data mining and machine learning technology. Therefore, how to realize the collection and analysis of user data on the basis of protecting user privacy is an important issue that needs to be studied.

[0005] In summary, how to apply LDP to data mining related algorithms and make full use of data to mine relevant knowledge under the premise of protecting user privacy has become a hot topic in current research. Summary of the invention

[0006] Purpose of the invention: In order to protect user personal privacy, the present invention provides a naive Bayes classification method under a local differential privacy model to address the application problem of naive Bayes classification under a local differential privacy model.

[0007] Technical solution: A naive Bayes classification method under a local differential privacy model, including the following steps:

[0008] (1) The server divides the user data set into a training data set and a classification prediction data set, and sets a privacy budget value ε; the data set consists of n user data, each of which includes several discrete features {A1, A2, ...A s} and a class label C;

[0009] (2) For the training data set, each user randomly samples a feature A r , the feature value x and the class label value y form a two-dimensional data (x, y), the class label value range size is l, the number of possible feature values ​​is t, and the perturbation processing is performed accordingly;

[0010] (3) Initialize the estimated frequency array of class labels And the estimated joint distribution array F = {0} for each feature and class label l×t After the server collects all the reported data from users, it performs statistical analysis on the data, estimates the frequency of each value of the class label, and saves it in an array And further estimate the joint distribution of each feature and class label, adding array The corresponding position of

[0011] (4) Initialize the final class label estimation result array The classification prediction dataset uses a naive Bayes classifier based on the estimation results in step (3) to predict the class label of each user.

[0012] Furthermore, the perturbation process in step (2) is to perturb the two-dimensional data locally. First, the value of the privacy budget is determined. If ε < 0.5, the UE-UE method is used to perturb (x, y). The other two-dimensional data is perturbed by the GRR-GRR method. Finally, a noise value after perturbation is obtained. Report it to the server.

[0013] The steps of perturbing (x, y) using the UE-UE method are as follows:

[0014] (S2.1.1) Encode y into an l-bit string, where only the index bit corresponding to the true value is 1, and the rest are 0;

[0015] (S2.1.2) Perturb each bit of y. If the initial value is 1, it remains unchanged with probability a, otherwise it is perturbed to 0. If the initial value is 0, it is perturbed to 1 with probability b, and the probability of 1-b remains unchanged at 0.

[0016] (S2.1.3) Based on the perturbation result of y, x is processed again. The perturbation of each bit in y includes the following four cases:

[0017] (a) 1→1: The feature value x is encoded as a t-dimensional bit string in the same way as the value y of the class label in (S2.1.1); if x is initially 1, it remains unchanged with probability p, otherwise it is flipped; if the initial bit is 0, it is perturbed to 1 with probability q, otherwise it remains unchanged;

[0018] (b) 1→0: Encode x into a t-dimensional bit string of all 0s;

[0019] (c) 0→1: Encode x into a t-dimensional bit string of all 0s; each bit is perturbed to 1 with probability q and remains unchanged with probability 1-q;

[0020] (d) 0→0: Encode x as a t-dimensional bit string of all zeros.

[0021] The GRR-GRR method is used to perturb two-dimensional data (x, y) including the following steps:

[0022] (S2.2.1) Perturb y to keep the user's true value unchanged with probability a, otherwise perturb to any other value in the value range;

[0023] (S2.2.2) Perturb x according to the perturbation result of y; if the value of y changes, then x remains unchanged with probability p, and a perturbation value is randomly sampled from the rest of the range of x with probability q;

[0024] (S2.2.3) The disturbed two-dimensional data Submit to the server.

[0025] Furthermore, step (3) includes the following processing steps:

[0026] (3.1) For each class label j∈[l], count the number of users who report this class label after perturbation, denoted by N j ;

[0027] (3.2) For each value of the class label, the perturbations of the class label in the UE-UE and GRR-GRR perturbation methods are analyzed to find that they satisfy the frequency estimation protocols UE and GRR in LDP respectively. The following calculation expression is used to estimate the frequency:

[0028]

[0029] (3.3) Calibrate the estimated frequency in step (3.2) to range and add it to the array The corresponding position in ;

[0030] (3.4) For each class label value j∈[l], analyze the joint distribution between j and each feature value k∈[t], and count the number of users who report class label j and feature value k after perturbation, denoted as c jk ;

[0031] (3.5) When perturbed according to the GRR-GRR method, the joint distribution is estimated according to the following formula;

[0032]

[0033] Otherwise, the estimation is performed as follows:

[0034]

[0035] (3.6) Calibrate the result in step (3.5) to the interval [0, 1] and add it to the array The corresponding position of .

[0036] Furthermore, the division of the privacy budget on the server side is determined by minimizing the mean square error of the joint distribution on the server side. In the UE-UE data perturbation processing, the actual privacy budget that can be used for the user's features is ε1=ε / 2, and the privacy budget that can be used for the class label is ε2=ε; in the GRR-GR data perturbation processing, the division of the privacy budget is ε2=ε.

[0037] In the method, the user locally uses the UE-UE or GRR-GRR method to perform data perturbation, and considers the relationship between features and class labels, rather than performing two-dimensional data perturbation alone.

[0038] Furthermore, in this method, the server divides the user data set into 80% training data set and 20% classification prediction data set.

[0039] Beneficial effects: The method described in the present invention is a method for estimating the joint distribution between discrete features and class labels, and finds a better way to divide the privacy budget of features and class labels. The designed scheme is used to estimate the joint distribution, and the value of the conditional probability is further obtained, and the category of the test record is estimated using the naive Bayes mechanism. Under the premise of satisfying local differential privacy, the classification prediction of the test data can still achieve a high accuracy. Compared with the prior art, the outstanding substantive features and significant improvements of the present invention are mainly reflected in the following points:

[0040] (1) Since each user has multiple features, if the user reports all the features, the privacy budget available for each feature will be very small, and the corresponding noise will be very large. In the present invention, each user randomly samples a feature for reporting, that is, the user division method can reduce noise and improve accuracy when the number of users is relatively large.

[0041] (2) Since there is a certain correlation between the user's features and the categories, the present invention takes the correlation between the data into consideration when perturbing the two data. The perturbation method of the features is closely related to the perturbation result of the class label.

[0042] (3) When perturbing the data, the LDP scheme is not used separately, so the privacy budget is no longer a simple addition. The present invention finds a better privacy budget allocation scheme through theoretical analysis of errors to improve the accuracy of the estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the overall flow chart of the present invention;

[0044] Figure 2 Schematic diagram of the disturbance scheme GRR-GRR method in the present invention. DETAILED DESCRIPTION

[0045] The present invention will be described or explained in a non-limiting manner by specific embodiments. The embodiments are only preferred solutions for applying the technical solutions of the present invention, and any technical solutions formed by equivalent replacement or equivalent transformation fall within the scope of protection claimed by the present invention.

[0046] The present invention provides a naive Bayes classification method under a local differential privacy model. UE and GRR are commonly used protocols for local differential privacy. The method proposed in the present invention includes UE-UE and GRR-GRR algorithms, which are new algorithms proposed based on the prior art. The specific implementation steps are as follows:

[0047] S1: There are n users in total, each of which has multiple discrete features {A1, A2, ...A s} and a class label C, all user data are aggregated into a dataset. The server first divides the user dataset into two parts, 80% of the data is used for training the model, and 20% of the data is used for classification prediction of the final class label, and sets a privacy budget value ε.

[0048] S2: For 80% of the data used for training, each user randomly samples a feature A from multiple features r , the feature value x and the class label value y form a two-dimensional data (x, y), the class label value range size is l, and the number of possible feature values ​​is t. Then use the two-dimensional data perturbation method proposed in this invention to perturb your own data locally, and the perturbed result is Report to the server.

[0049] The perturbation steps are:

[0050] S2.1: Determine the value of the privacy budget. If ε < 0.5, use the UE-UE method in the present invention to perturb (x, y).

[0051] S2.1.1: Encode y into an l-bit string. Only the index bits corresponding to the true value are 1, and the rest are 0.

[0052] S2.1.2: Perturb each bit of y. If the initial value is 1, keep it unchanged with probability a, otherwise perturb it to 0; if the initial value is 0, perturb it to 1 with probability b, and keep it unchanged with probability 1-b.

[0053] S2.1.3: Process x based on the perturbation result of y. According to S2.1.2, there are four cases for the perturbation of each bit in y:

[0054] (a) 1→1: The feature value x is encoded as a t-dimensional bit string in the same way as the value y of the class label in S2.1.1. If x is initially 1, it remains unchanged with probability p, otherwise it is flipped; if the initial bit is 0, it is perturbed to 1 with probability q, otherwise it remains unchanged.

[0055] (b) 1→0: Encode x as a t-dimensional bit string of all zeros.

[0056] (c) 0→1: Encode x into a t-dimensional bit string of all 0s. Each bit is perturbed to 1 with probability q and remains unchanged with probability 1-q.

[0057] (d)0→0: The perturbation process is the same as (b).

[0058] S2.1.4: The perturbed two-dimensional data Submit to the server.

[0059] S2.2: Otherwise, the GRR-GRR method is used to perturb the user's own two-dimensional data (x, y).

[0060] S2.2.1: Perturb y to keep the user's true value unchanged with probability a, otherwise perturb to any other value in the value range.

[0061] S2.2.2: Perturb x according to the perturbation result of y; if the value of y changes, then x remains unchanged with probability p, and a perturbation value is randomly sampled from the rest of the range of x with probability q.

[0062] S2.2.3: The perturbed two-dimensional data Submit to the server.

[0063] S2.3: Finally, a perturbed noise value is obtained Report it to the server.

[0064] S3: Initialize the estimated frequency array of class labels And the estimated joint distribution array F = {0} for each feature and class label l×t After the server collects all the user-reported data, it performs statistical analysis on the data, estimates the frequency of each value of the class label, and saves it in an array In the above example, the joint distribution of each feature and class label is further estimated by adding the number of red The corresponding position of .

[0065] S3.1: For each class label j∈[l], count the number of users who report this class label after perturbation, denoted by N j .

[0066] S3.2: For each value of the class label, analysis of the perturbations of the class label in the UE-UE and GRR-GRR perturbation methods shows that they satisfy the frequency estimation protocols UE and GRR in LDP respectively, so the frequency can be estimated using the following formula.

[0067]

[0068] S3.3: Calibrate the estimated frequency in S3.2 to range and add it to the array The corresponding position in .

[0069] S3.4: For each class label value j∈[l], analyze the joint distribution between j and each feature value k∈[t]. Count the number of users who report class label j and feature value k after the perturbation, denoted as c jk .

[0070] S3.5: When perturbed according to the GRR-GRR method, the joint distribution is estimated according to the following formula;

[0071]

[0072] Otherwise, the estimation is performed as follows:

[0073]

[0074] S3.6: Calibrate the result in step S3.5 to the interval [0, 1] and add it to the array The corresponding position of .

[0075] S4: Initialize the final class label estimation result array The remaining 20% ​​of the data is used for prediction. Based on the estimation results in S3, a naive Bayes classifier is used to predict the class label of each user.

[0076] Combination Figure 1 The present invention describes a naive Bayes classification method based on a local differential privacy model, including processing processes on the user side and the server side.

[0077] The steps that the user needs to perform are as follows:

[0078] Step 1: Each user has s discrete features {A1, A2, ...A s}, and its value x and the value y of the class label C form a two-dimensional data (x, y). The value ranges of x and y are X and Y respectively, and the number of possible values ​​in X and Y is t and l.

[0079] Step 2: Obtain the value of the privacy budget set by the server and the division of the privacy budget. Determine the value of the privacy budget. If ε < 0.5, use the UE-UE method in the present invention to perturb (x, y) locally. First, encode and perturb y in the following way, where j∈[l]

[0080]

[0081] According to the perturbation results of y, x can be perturbed in the following four cases.

[0082] (a) Encode and perturb x as follows. Where k∈[t]

[0083]

[0084] (b) Directly encode x as a binary vector with all bits set to 0.

[0085] (c) Encode and perturb x as follows. Where k∈[t]

[0086]

[0087] (d) Same operation as (b).

[0088] Step 3: Otherwise, use the GRR-GRR method to perturb the user's own two-dimensional data (x, y). Figure 2 , first perturb y according to the following probability, the calculation process is as follows:

[0089]

[0090] if Then x is perturbed with the following probability, otherwise a value is randomly sampled from the range X of x as the perturbation value.

[0091]

[0092] Step 4: Finally get a perturbed noise value Report it to the server.

[0093] The steps required on the server side are as follows:

[0094] Step 1: There are n users in total, each of which has multiple discrete features {A1, A2, ...A s} and a class label C, all user data are aggregated into a dataset. The server first divides the user dataset into two parts, 80% of the data is used for training the model, and 20% of the data is used for classification prediction of the final class label, and sets a privacy budget value ε.

[0095] Step 2: Minimize the mean square error of the joint distribution to obtain a better way to divide the privacy budget.

[0096] (1) In UE-UE, the actual privacy budget available for user features is ε1=ε / 2, and the privacy budget available for class labels is ε2=ε.

[0097] (2) In GRR-GRR, the privacy budget is divided as follows:

[0098] Step 3: After the server collects 80% of the data reported by all users in the training set, it performs statistical analysis on the data. Initialize the estimated frequency array of the class label And the estimated joint distribution array F = {0} for each feature and class label l×t .

[0099] Step 4: For each class label j∈[l], count the number of users who report this class label after perturbation, denoted as N j .

[0100] The following formula is used to estimate the frequency of each value of the class label and calibrate it to Range, stored in array middle.

[0101]

[0102] Step 5: For each class label value j∈[l], analyze the joint distribution between j and each feature value k∈[t]. Count the number of users who report class label j and feature value k after the perturbation, denoted as c jk .

[0103] The joint distribution of each feature and class label is further estimated.

[0104] 1) When perturbed by the GRR-GRR method, the joint distribution is estimated according to the following formula;

[0105]

[0106] 2) When perturbing according to the UE-UE method, the joint distribution is estimated according to the following formula:

[0107]

[0108] Calibrate the estimated joint distribution results to the interval [0, 1] and add them to the array The corresponding position of .

[0109] Step 6: Initialize the final class label estimation result array For the remaining 20% ​​test data, according to the estimation results in S3, we first need to calculate the conditional probability based on the frequency of the class label and the estimated joint distribution. Since the estimated joint distribution may be 0, we set 0 to 0.001. Finally, we substitute it into the naive Bayes classifier to predict the class label of each user.

[0110]

[0111]

[0112] The following are the experimental results of the naive Bayes classification method under the local differential privacy model. The experiment uses the Nursery dataset in UCl, which contains 12,960 instances, 8 features, and the value range of the class label is 5. The data scale is 5×5. In the experiment, the privacy budget is set to 0.2, 1, 2, 3, 4, 5, and 6 respectively. The Euclidean distance is used as the measurement standard. Finally, the error of the joint distribution between each feature and the class label is shown in Table 1, and the accuracy of the naive Bayes classification is shown in Table 2.

[0113] According to the experimental results, as the privacy budget continues to increase, the error of the joint distribution estimated by the present invention is also continuously reduced. Finally, the accuracy of the naive Bayes classification has been increasing, and can eventually reach an accuracy of about 90%. It is achieved on the basis of protecting the number of users and also ensuring the accuracy of data analysis.

[0114] Table 1 Error table of estimated joint distribution and true joint distribution

[0115]

[0116]

[0117] Table 2 Naive Bayes classification accuracy table

[0118]

Claims

1. A naive Bayes classification method under a local differential privacy model, characterized by: The following steps are involved: (1) The server divides the user data set into a training data set and a classification prediction data set, and sets a privacy budget value ε; the data set consists of n user data, each of which includes several discrete features {A1, A2, ...A s } and a class label C; (2) For the training data set, each user randomly samples a feature A r , the value x of the feature and the value y of the class label form a two-dimensional data (x, y), the value range of the class label is l, the number of possible values ​​of the feature is t, and the perturbation processing is performed accordingly; Perturbation processing is to perturb the two-dimensional data locally. First, determine the value of the privacy budget. If ε<0.5, use the UE-UE method to perturb (x, y). For other two-dimensional data, use the GRR-GRR method to perturb the user's own two-dimensional data (x, y). Finally, a perturbed noise value is obtained. Report it to the server; The GRR-GRR method is used to perturb two-dimensional data (x, y) including the following steps: (S2.2.1) Perturb y to keep the user's true value unchanged with probability a, otherwise perturb to any other value in the value range; (S2.2.2) Perturb x according to the perturbation result of y; if the value of y changes, then x remains unchanged with probability p, and a perturbation value is randomly sampled from the rest of the range of x with probability q; (S2.2.3) The disturbed two-dimensional data Submit to the server; (3) Initialize the estimated frequency array of class labels And the estimated joint distribution array F = {0} for each feature and class label l×t After the server collects all the reported data from users, it performs statistical analysis on the data, estimates the frequency of each value of the class label, and saves it in an array And estimate the joint distribution of each feature and class label, adding array The corresponding position of (4) Initialize the final class label estimation result array The classification prediction dataset uses the naive Bayes classifier based on the estimation results in step (3) to predict the class label of each user.

2. The naive Bayes classification method under the local differential privacy model according to claim 1, characterized in that: The steps of perturbing (x, y) using the UE-UE method are as follows: (S2.1.1) Encode y into an l-bit string, where only the index bit corresponding to the true value is 1, and the rest are 0; (S2.1.2) Perturb each bit of y. If the initial value is 1, it remains unchanged with probability a, otherwise it is perturbed to 0. If the initial value is 0, it is perturbed to 1 with probability b, and the probability of 1-b remains unchanged at 0. (S2.1.3) Based on the perturbation result of y, x is processed again. The perturbation of each bit in y includes the following four cases: (a) 1→1: Encode the feature value x into a t-dimensional bit string; if x is initially 1, it remains unchanged with probability p, otherwise it is flipped; if the initial bit is 0, it is perturbed to 1 with probability q, otherwise it remains unchanged; (b) 1→0: Encode x into a t-dimensional bit string of all 0s; (c) 0→1: Encode x into a t-dimensional bit string of all 0s; Each bit is perturbed to 1 with probability q and remains unchanged with probability 1-q; (d) 0→0: Encode x as a t-dimensional bit string of all zeros.

3. The naive Bayes classification method under the local differential privacy model according to claim 1, characterized in that: Step (3) includes the following processing steps: (3.1) For each class label j∈[l], count the number of users who report this class label after perturbation, denoted by N j ; (3.2) For each value of the class label, based on the UE-UE and GRR-GRR perturbations, the frequency estimation protocols UE and GRR in LDP are satisfied respectively, and the frequency is estimated using the following calculation expression: a＝1 / 2， In the above formula, ε1 is the privacy budget that can actually be used by the user's features; (3.3) Calibrate the estimated frequency in step (3.2) to range and add it to the array The corresponding position in ; (3.4) For each class label value j∈[l], analyze the joint distribution between j and each feature value k∈[t], and count the number of users who report class label j and feature value k after perturbation, denoted as c jk ; (3.5) When perturbed according to the GRR-GRR method, the joint distribution is estimated according to the following formula; Otherwise, the estimation is performed as follows: Where ε2 is the privacy budget available for class labels; (3.6) Calibrate the result in step (3.5) to the interval [0,1] and add it to the array The corresponding position of .

4. The naive Bayes classification method under the local differential privacy model according to claim 1, characterized in that: The division method of the privacy budget on the server side is determined by minimizing the mean square error of the joint distribution on the server side. In the UE-UE data perturbation processing, the actual privacy budget that can be used for the user's features is ε1=ε / 2, and the privacy budget that can be used for the class label is ε2=ε; in the GRR-GR data perturbation processing, the division method of the privacy budget is ε2=ε.

5. The naive Bayes classification method under the local differential privacy model according to claim 1, characterized in that: In the method, the user locally uses the UE-UE or GRR-GRR method to perform data perturbation, and considers the relationship between features and class labels, rather than performing two-dimensional data perturbation alone.

6. The naive Bayes classification method under the local differential privacy model according to claim 1, characterized in that: The server divides the user data set into 80% training data set and 20% classification prediction data set.

Images