An adaptive histogram publishing method for data stream sliding window
By combining the sliding window approximate statistical counting and online noise addition strategy, adaptively publishing the histogram of the data flow sliding window, solving the problems of low availability and high spatiotemporal overhead in the prior art, and achieving efficient data processing and Data protection.
Patent Information
- Application Number
- CN202111222014.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-20
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-10-20
AI Technical Summary
When processing data flow sliding windows, the prior art has problems such as low availability of publishing data and high spatiotemporal overhead, and it has failed to effectively combine data flow approximate statistical methods and data privacy protection.
Using sliding window approximate statistical counting method and optimized online noise addition strategy, the histogram of histogram is released by allocating privacy budgets, and the data flow histogram information at the next moment is predicted by sliding window approximate statistical statistics, and the difference between the estimated value and the real value is compared. Select the appropriate noise value, optimize the data error in the group, and build an approximate statistical histogram of the current time.
Improves data availability, significantly reduces running time, and improves data security while reducing privacy budgets, reduces memory consumption, and is suitable for environments with harsh hardware conditions.
Smart Images

Figure CN113934772B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information security technology, and more specifically, to an adaptive histogram publishing method for a data stream sliding window. Background Art
[0002] With the rapid development of information technology, many applications often need to dynamically publish the data they have generated or acquired. For example: (1) When users browse websites, they can obtain the user's browsing information and analyze the user's preferences; (2) In the real-time traffic information system, the current traffic conditions are analyzed to help plan the destination and predict the traffic status; (3) The medical and health system needs to publish the patient's medical data in real time.
[0003] As one of the most effective standards and methods for data privacy protection, differential privacy has been widely used in many fields such as disease monitoring, real-time traffic monitoring, and Internet traffic analysis.
[0004] Essentially, the principle of differential privacy to protect sensitive data is to add random noise to the data to achieve the purpose of protecting sensitive data. Note that, on the one hand, after adding random noise, when the user queries the data in the database, the query results obtained will contain these randomly added noise. This makes it impossible for users to reverse the query results to obtain sensitive data information in the database. On the other hand, these added noises also reduce the accuracy of the original data and affect the usefulness of the data (such as the query results are too distorted). Therefore, when using the differential privacy algorithm, in order to achieve the required data protection level and ensure the usefulness of the data, it is necessary to strictly control the scale of noise addition - that is, if too much noise is added, the protection level of sensitive data will be enhanced, but the usefulness of the data will be reduced; conversely, if too little noise is added, the usefulness of sensitive data will be enhanced, but the protection level of the data will be reduced.
[0005] At present, although there are many histogram publishing methods for static data sets, these methods need to cache a large amount of historical data when processing data streams and can hardly process data streams quickly (long running time). On the other hand, although there are some histogram publishing methods specifically for data stream sliding windows, these methods have the following problems:
[0006] (1) Existing methods do not consider the inherent correlation between data stream approximate statistical methods and data privacy protection issues, and do not use this feature to design solutions to the problem. More specifically: data stream approximate statistical methods can generally quickly return an approximate estimate of a true value x. because It can also be regarded as a kind of privacy protection for x. The basic protection mechanism of differential privacy is also to protect x by adding a noise variable ε to the true value x. Therefore, there is a close correlation between the two.
[0007] (2) Existing methods have high running time and space overhead. More specifically, existing differential privacy methods still rely on the design strategy of caching the entire window data at all times and then adding noise to it and publishing it. This requires these algorithms to consume a large amount of memory space to store historical data, so as to ensure that histograms can be created at any time. At the same time, since the histograms generated by these algorithms at each moment are obtained by adding noise to all the data in the window one by one, their time overhead is also high.
[0008] Chinese Patent Application No. 202011045992.1, Application Publication Date: February 2, 2021, Patent Name: Data Stream Differential Privacy Histogram Publishing Method Based on Sliding Window, the patent first divides a sliding window into k sub-blocks based on the idea of data block, and uses this parameter to control and adjust the statistical error of the data histogram; then, the privacy budget allocation of the current sliding window is optimized by comparing the difference in the data distribution of two adjacent histograms, so as to quickly calculate the local optimal histogram; the method designed by the patent meets the requirements of differential privacy, and its approximate error does not exceed W / 2k. However, the technical solution provided by the patent is to directly make an approximate estimate of the data in each sliding window, and then directly add Laplace noise to these data. The patent only uses a privacy budget ε to protect the privacy of approximate data, which cannot be applied to the group-based histogram publishing method. Summary of the invention
[0009] 1. Technical problem to be solved by the invention
[0010] In view of the above shortcomings of the prior art, the present invention provides an adaptive histogram publishing method for a data stream sliding window. The present invention combines the sliding window approximate statistical counting method and the optimized online noise addition strategy to adaptively publish the histogram to solve the problems of low availability and high time and space overhead of the existing published data. The present invention can efficiently process the data in the data stream and achieve a publishing error that satisfies the user.
[0011] 2. Technical solution
[0012] In order to achieve the above object, the technical solution provided by the present invention is:
[0013] The present invention provides an adaptive histogram publishing method for a data stream sliding window, the steps of which are as follows:
[0014] Step 1: Determine the interval of the noise histogram to be published;
[0015] Step 2: Divide the overall privacy budget ε into two parts. The first part of the privacy budget is αε, which is used to add noise to the data before sorting. The second part of the privacy budget is ε(1-α), which is used to add noise to the histogram greedy algorithm after sorting.
[0016] Step 3: For each interval in the published histogram, use the sliding window approximate statistics to predict the estimated information of the data stream histogram at the next moment, compare the privacy budget plus noise value of the first part with the approximate statistical value, select a suitable noise value, cluster the sorted histogram intervals and optimize the error of the data within the group, and construct the approximate statistical histogram at the current moment;
[0017] Step 4: For the histogram data selected at the current moment, add the noise value of the second part of the privacy budget to the data through the error lower limit function;
[0018] Step 5: Publish the noisy histogram of the sliding window at the current moment.
[0019] Furthermore, in step 2, a small sample is used for preprocessing, and the optimal privacy budget allocation factor α is obtained by adjusting different privacy budget allocation factors α. The difference between the noisy histogram obtained at the current moment in the small sample and the true value at the current moment is used to calculate.
[0020] Furthermore, in step 3, the calculation process of using sliding window approximate statistics to predict the data stream histogram estimation information at the next moment is:
[0021] 1) Determine whether the current data entering the sliding window data stream meets the determined statistical interval; if the current data meets the current interval range, the current data is set to 1; if the current data does not meet the current interval range, the current data is set to 0;
[0022] 2) Check the position where the current data enters the sub-block: if the current data is not at the end of the sub-block, the block offset m of the current sub-block is increased by 1, and the value of the counter y is increased by 1; if the current data is at the end of the sub-block, the next data enters the next sub-block, and the sub-block offset m is reset to 0. When the value of the counter y increases by 1 and exceeds the size of the sub-block, the size of the sub-block is subtracted from the value of the counter y and the count is restarted. At the same time, the number of bits in the array b corresponding to the number of times the value of the counter y exceeds the size of the sub-block is set to 1.
[0023] 3) Output the approximate statistical value at the current time i
[0024] The size of the sliding window is W, the sliding window is divided into k sub-blocks, the size of the sub-block is W / k, the array of k bits is b, B is the statistic whose median of array b is 1, m*b i is the expired data of the current sliding window, 2Wk is the error between the approximate statistical frequency of the interval and the true frequency;
[0025] 4) Judgment All results less than 0 are set to 0 to get the latest
[0026] Furthermore, in step 3, the privacy budget plus noise value of the first part is compared with the approximate statistical value, and the calculation process of selecting the appropriate noise value is as follows:
[0027] 1) Calculate the approximate statistical value of the current sliding window data through approximate statistics
[0028] 2) Add noise using the difference between the approximation error and the noise threshold;
[0029] 3) Repeat the above steps for all intervals at the current time to obtain or In the judgment interval or All results less than 0 are set to 0 to obtain the statistical values of all intervals in the current window;
[0030] Furthermore, the specific process of adding noise using the difference between the approximate error and the noise threshold is as follows:
[0031] (a) If the approximation error is less than the noise threshold, use the approximation Replace the noise value
[0032] (b) If the noise threshold is smaller than the approximation error, use the noise value Replace approximation
[0033] Among them, the approximate error is the approximate statistical result With the true value H i The difference between the noise threshold and the noise value With the true value H i The maximum difference between the two, the noise threshold in the present invention is
[0034] Furthermore, in step 3, clustering is performed on the sorted histogram intervals and the error of the data within the group is optimized. The calculation process of constructing the approximate statistical histogram at the current moment is:
[0035] 1) The value selected in the current sliding window Insert into current histogram statistics;
[0036] 2) First, use the sorting algorithm to sort the statistical data in the sliding window; and calculate the error between each frequency and other frequencies when combined and the error when not combined;
[0037] 3) Determine the interval grouping;
[0038] a) When the error of merging two intervals is less than the error of not merging, the two intervals are merged and set as one group;
[0039] b) When the error of merging two intervals is greater than the error of not merging, the two intervals are not merged and are set as two groups;
[0040] Among them, the merged error is the result of merging the intervals and taking the square difference between the average value of each interval and the existing value of each interval; the unmerged error is the result of not merging the intervals and taking the square difference between the average value of each interval and the existing value of each interval;
[0041] 4) Group the data in the current sliding window using the above interval grouping method;
[0042] 5) Use the average value of each group to replace the frequency value in each group, and determine the privacy budget in each interval by the proportion of the statistical value in each group to the total number of data statistical values in the entire sliding window to add Laplace noise;
[0043] 6) In order to reduce the error caused by approximate statistics, judge All results less than 0 are set to 0, and the values that do not meet the conditions are filtered out. Finally, the noise histogram at the current time i is returned.
[0044] Furthermore, in step 4, the error lower limit function is used to obtain the noised histogram of the current moment by adding the second part of the privacy budget to the histogram data:
[0045] The first interval of the sorted approximate statistical histogram forms a group by itself, and the noise error introduced by the second interval after being added to the current group and the noise error introduced by the new group re-established with the second interval as the starting point are judged; if the noise error introduced by adding the current group is smaller than that introduced by establishing a new group, then add the current group, otherwise establish a new group; according to this rule, add noise to all histograms.
[0046] A computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the above-mentioned adaptive histogram publishing method for a data stream sliding window is implemented.
[0047] 3. Beneficial effects
[0048] Compared with the existing known technologies, the technical solution provided by the present invention has the following significant effects:
[0049] (1) The present invention is an adaptive histogram publishing method for a data stream sliding window. First, the data stream approximate count is used to predict the distribution information of the data in the sliding window at the next moment; then the appropriate publishing value is selected by comparing the difference between the estimated value and the true value; finally, the sorted histogram intervals are clustered and the error of the data within the group is optimized. The data availability is effectively improved and the running time is significantly reduced.
[0050] (2) The adaptive histogram publishing method for data stream sliding window of the present invention can protect some important data and increase data security. When processing large amounts of data, it can effectively improve data availability and reduce running time while reducing privacy budget. It can also reduce memory consumption and hardware requirements of software, and can be used in places with harsh hardware conditions. Through experiments on real data sets, the experimental results on two different real data sets also verify that the present invention is significantly better than existing methods in terms of data availability and running time. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 Schematic diagram of the privacy budget allocation results of the APS algorithm;
[0052] Figure 2 Schematic diagram of the impact of different privacy budgets ε on availability; Figure 2 (a) shows the different privacy budget availability results of the UK car accident dataset. Figure 2 (b) shows the availability results of different privacy budgets for the New York yellow taxi dataset;
[0053] Figure 3 Schematic diagram of the impact of different window block k on availability; where, Figure 3 (a) shows the availability results of different window blocks of the UK traffic accident dataset. Figure 3 (b) shows the availability results of different window blocks for the New York yellow taxi dataset;
[0054] Figure 4 Schematic diagram of the impact of different sliding windows W on availability; Figure 4 (a) shows the availability results of different sliding window sizes for the UK car accident dataset. Figure 4 (b) shows the availability results of different sliding window sizes for the New York yellow taxi dataset;
[0055] Figure 5 Schematic diagram of the impact of sliding window W on total running time (New York yellow taxi trip data set);
[0056] Figure 6 Schematic diagram of the impact of the number of sliding window blocks k on the total running time (New York yellow taxi trip data set);
[0057] Figure 7 This is the processing flow chart of the data stream differential privacy histogram algorithm;
[0058] Figure 8 Algorithm 1 is based on sliding window adaptive histogram publishing method APS;
[0059] Fig. 9 Algorithm 2 is an adaptive denoising method ANM based on sliding window approximate estimation. DETAILED DESCRIPTION
[0060] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments.
[0061] Example 1
[0062] In the prior art, although there are many histogram publishing methods on static data sets, there are few histogram publishing methods based on sliding windows in data stream environments, and they face the problem of high histogram publishing errors. In view of this, this embodiment provides an adaptive histogram publishing method for data stream sliding windows. This embodiment further optimizes the histogram publishing technology under differential privacy conditions by combining sliding windows and differential privacy technologies. The existing method needs to add uniform noise to the data histogram information to achieve data protection, which leads to problems such as low data availability and high time complexity in practical applications.
[0063] This embodiment performs a single-pass scan on a given data stream D and uses a sliding window model for statistical publishing. First, the approximate count of the data stream is used to predict the distribution information of the data in the sliding window at the next moment, and then the appropriate publishing value is selected by comparing the difference between the estimated value and the true value; finally, the sorted histogram intervals are clustered and the error of the data within the group is optimized.
[0064] More specifically, this embodiment includes the following steps:
[0065] Step 1: Determine the interval of the noise histogram to be published;
[0066] Step 2: Divide the overall privacy budget ε into two parts. The first part of the privacy budget is αε, which is used to add noise to the data before sorting. The second part of the privacy budget is ε(1-α), which is used to add noise to the histogram after sorting using a greedy algorithm. The calculation process for the optimal privacy budget allocation factor α is:
[0067] For the noisy statistical histogram obtained at the current moment, the present invention uses a small sample for preprocessing, calculates by adjusting different privacy budget allocation factors α, and obtains an optimal privacy budget allocation factor α through the difference between the noisy histogram obtained at the current moment in the small sample and the true value at the current moment.
[0068] Step 3: For each interval in the published histogram, the data stream data in the sliding window model at the current moment uses an approximate statistical algorithm to obtain the approximate statistical value of the current interval, and compares the first part of the privacy budget plus noise value with the approximate statistical value, selects a suitable noise value, filters the intervals with approximate statistical frequencies less than 0, and then constructs the approximate statistical histogram at the current moment according to the approximate statistical frequencies of each interval. The details are as follows:
[0069] (1) Using approximate counting of data streams to predict the distribution information of data in the sliding window at the next moment; including the following steps:
[0070] Define the size of the sliding window as W, the data stream input into the sliding window as D, based on the idea of data block, divide the sliding window into k sub-blocks, the size of the sub-block is W / k, a k-bit array is set to b, the offset of the first sub-block that the data stream D enters for the first time is m, and the counter of 1 is y;
[0071] 1) Determine whether the current data entering the sliding window data stream meets the determined statistical interval; if the current data meets the current interval range, the current data is set to 1; if the current data does not meet the current interval range, the current data is set to 0;
[0072] 2) Check the position where the current data enters the sub-block: if the current data is not at the end of the sub-block, the block offset m of the current sub-block is increased by 1, and the value of the counter y is increased by 1; if the current data is at the end of the sub-block, the next data enters the next sub-block, and the sub-block offset m is reset to 0. When the value of the counter y increases by 1 and exceeds the size of the sub-block, the size of the sub-block is subtracted from the value of the counter y and the count is restarted. At the same time, the number of bits in the array b corresponding to the number of times the value of the counter y exceeds the size of the sub-block is set to 1.
[0073] 3) Output the approximate statistical value at the current time i
[0074] Among them, B is the statistic whose median of array b is 1, m*b i is the expired data of the current sliding window, is the error threshold between the approximate statistical frequency of the interval and the true frequency.
[0075] 4) In order to reduce the error caused by approximate statistics, judge All results less than 0 are set to 0 to get the latest (2) Compare the difference between the estimated value and the true value to select the appropriate release value; including the following steps:
[0076] 1) Calculate the approximate result of the current sliding window data through the approximate statistics in step 1
[0077] 2) Add noise using the difference between the approximate error and the noise threshold.
[0078] (a) If the approximation error is less than the noise threshold, use the approximation Replace the noise value
[0079] (b) If the noise threshold is smaller than the approximation error, use the noise value Replace approximation
[0080] Among them, the approximate error is the approximate statistical result With the true value H i The difference between
[0081] 3) Repeat the above steps for all intervals at the current time to obtain or In order to reduce the error caused by approximate statistics, the or All results less than 0 are set to 0, and finally all interval statistics in the current window are obtained.
[0082] (3) Clustering the sorted histogram intervals and optimizing the errors of the data within the group; including the following steps:
[0083] 1) The value selected in the current sliding window Insert into the current histogram statistics.
[0084] 2) First, use the sorting algorithm to sort the statistical data in the sliding window; and calculate the combined error and the uncombined error between each frequency and other frequencies.
[0085] 3) Determine the interval grouping.
[0086] a) When the error of merging two intervals is less than the error of not merging, the two intervals are merged and set as one group.
[0087] b) When the error of merging two intervals is greater than the error of not merging them, the two intervals are not merged and are set as two groups.
[0088] The merged error is the result of merging the intervals and taking the square difference between the average value of each interval and the existing value of each interval. The unmerged error is the result of not merging the intervals and taking the square difference between the average value of each interval and the existing value of each interval.
[0089] 4) Group the data in the current sliding window using the above interval grouping method.
[0090] 5) Use the average value of each group to replace the frequency value in each group. The privacy budget in each interval is determined by the proportion of the statistical value in each group to the total number of data statistical values in the entire sliding window to add Laplace noise.
[0091] 6) In order to reduce the error caused by approximate statistics, judge All results less than 0 are set to 0, and the values that do not meet the conditions are filtered out. Finally, the noise histogram at the current time i is returned.
[0092] Step 4: For the selected histogram data at the current moment, add the corresponding noise value of the second part of the privacy budget to these data through the error lower limit function. The specific process is:
[0093] The first interval of the approximate statistical histogram after sorting is a group by itself. The noise error introduced by the second interval after adding it to the current group and the noise error introduced by the new group re-established with the second interval as the starting point are judged; if the noise error introduced by adding it to the current group is smaller than that introduced by establishing a new group, then add it to the current group, otherwise establish a new group. According to this rule, add noise to all histograms.
[0094] Step 5: Publish the noisy histogram of the sliding window at the current moment.
[0095] Privacy Analysis
[0096] The APS algorithm satisfies ε-differential privacy.
[0097] Proof: Line 2 of Algorithm 1 uses the Laplace mechanism to compute the noise count for each bin. Line 2 satisfies ε1-differential privacy. Using Property 2 to explain, Line 9 of Algorithm 1 uses the Laplace mechanism to compute the noise mean for each cluster. Since the size of each group is known, this is equivalent to asking for the sum of the noise for each group. Line 9 satisfies ε2-differential privacy. Using Property 1 to explain, we know that Line 2 and Line 9 together satisfy (ε1+ε2)-differential privacy. This means that the APS algorithm satisfies ε-differential privacy.
[0098] Space complexity analysis
[0099] (1) The ANM algorithm requires k+2logW+O(1) bits of memory.
[0100] Proof: We use bit, m used bits and b bits use k bits. In addition, i needs bit, B needs another In general, the number of bits required for the ANM algorithm is
[0101] (2) The APS algorithm requires Mk+2MlogW+O(1) bits of memory.
[0102] Proof: The space complexity of the APS algorithm is determined by two parts. The first part is the adaptive noise addition method based on sliding window approximate estimation; the second part is the grouping method of greedy clustering. The space complexity of the APS algorithm is mainly determined by the data structure in the adaptive noise addition method based on sliding window approximate estimation. The memory required by the ANM algorithm is k+2logW. For a histogram with M intervals, the space required by the APS algorithm is Mk+2MlogW+O(1).
[0103] In the following, (1) UK Accidents extracts detailed road safety data on personal injury road accidents in the UK from 2005 to 2014. Each record contains attributes such as age, gender, severity, social class, and type of casualty, where the age range is [0,100]. (2) Taxi extracts a detailed trip data set of New York yellow taxis in 2019. Each record includes attributes such as pick-up date / time, pick-up location, trip distance, rate type, and driver-reported passenger count, where the passenger count range is [0,6] and is published as a histogram over time.
[0104] Combination Figure 1In the comparative analysis of the embodiment of the influence of mean square error under different allocation privacy budget ratios shown in the figure, the results show that: the mean square error first decreases with the increase of the privacy budget allocation ratio α; then, when the privacy budget allocation ratio α is 0.5, the mean square error reaches the lowest point; finally, the mean square error increases with the increase of the privacy budget allocation ratio α. There are two reasons for this: (1) In the adaptive noise addition method, a suitable value is selected between the approximate statistical value and the noise value, so the algorithm can increase the availability of published data while improving the operating efficiency of the data; (2) The privacy budget in the greedy grouping algorithm is mainly used to add Laplace noise. As the privacy budget increases, the less noise is added to the data, the smaller the final data error. Among them, the mean square error is most suitable when the privacy budget allocation ratio α is 0.5. Therefore, in the experiments later in this embodiment, the privacy budget allocation ratio α is set to 0.5.
[0105] Combination Figure 2 In the comparative analysis of the effects of different privacy budgets ε on availability shown in the example, the results show that as the privacy budget increases, the data has less mean square error, making the data availability higher. The algorithm in this paper (APS algorithm for short) uses an adaptive noise addition method (the difference between the predicted value and the true value is added with the same threshold) to result in lower error and better availability than the APB algorithm and the DiffHR algorithm.
[0106] Among them, the DiffHR algorithm (differentially private histogram release) is an accurate histogram release method that satisfies differential privacy, and the APB algorithm (adaptive privacy budget allocation) is an adaptive privacy budget allocation strategy based on static data.
[0107] Combination Figure 3 As shown in the comparative analysis of the embodiments of the impact of different window block k on availability, the results show that among the three methods, only the mean square error of the APS algorithm decreases with the increase of block k. The reason is that as the block k increases, the approximate statistical value will change less, resulting in a decrease in the total error. When the number of sliding block windows is too small, the APS algorithm and the APB algorithm have the same mean square error. The mean square error of the APB algorithm and the DiffHR algorithm has no direct relationship with the block size because the sliding window approximate statistical error is not used. The mean square error of the APS algorithm is significantly lower than that of the APB algorithm and the DiffHR algorithm because the APS algorithm uses the comparison between the sliding window approximate statistical value and the noise value, thereby avoiding the larger error caused by directly adding noise to the sliding window.
[0108] Combination Figure 4In the comparative analysis of the embodiment of the effect of different sliding window W sizes on availability shown in the figure, the results show that the reason for the increase in the mean square error of the APS algorithm is that the sliding window size increases. As the sliding window becomes larger, the comparison between the Laplace error and the approximate error adds data, which leads to an increase in the total error and an increase in the mean square error. The mean square error of the APS algorithm is significantly less than that of the APB algorithm and the DiffHR algorithm. The reason is that the APS algorithm uses sliding window approximate statistical values instead of noise values, thereby avoiding the result of directly adding noise to the sliding window. When the sliding window reaches a certain value, the APS algorithm and the APB algorithm have the same mean square error.
[0109] Combination Figure 5 In the comparative analysis of the embodiment of the influence of different sliding window W sizes on the total running time shown in the figure, the results show that the total running time of the APS algorithm, the DiffHR algorithm and the APB algorithm increases with the increase of the sliding window size W. This is because as the sliding window increases, the data processed by the three algorithms will increase, resulting in an increase in the total running time. Among them, the total running time of the APB algorithm is higher than that of the APS algorithm and the DiffHR algorithm. This is because the APB algorithm uses an optimization model for the allocation weights of the privacy budget, which leads to an increase in the total running time. When the sliding window size W and the sliding window block k are equal, the APS algorithm needs to cache all data in the current window, so that the APS algorithm and the DiffHR algorithm have the same running time. As the sliding window size W increases, the APS algorithm uses an approximate statistical algorithm to improve the efficiency of the algorithm, resulting in the total running time of the APS algorithm being less than that of the DiffHR algorithm.
[0110] Combination Figure 6 In the comparative analysis of the embodiment of the effect of different block window k sizes on the total running time shown, the results show that the total running time of the DiffHR algorithm and the APB algorithm are different constant times, regardless of k. This is because both algorithms need to cache the data in the entire sliding window at every moment, and because the APB algorithm uses an optimization model for the privacy budget allocation weights, the total running time increases. The total running time of the APS algorithm increases with the increase of the sliding window block k. This is because the ANM algorithm uses an approximate statistical algorithm, which makes the time overhead less than the DiffHR algorithm and the APB algorithm. When the sliding window size W is less than or equal to the number of blocks k, the ANM algorithm uses an exact count, resulting in the APS algorithm and the DiffHR algorithm having the same total running time.
[0111] The adaptive histogram publishing method APS for the data stream sliding window model disclosed in this embodiment can quickly publish the histogram of private data in a data stream environment. This method can quickly publish histogram data in a data stream environment at a low space cost. Theoretical analysis shows that this embodiment can effectively improve data availability and reduce running time while reducing the privacy budget. Experimental results on two different real data sets also verify that the present invention is significantly better than existing methods in terms of data availability and running time.
[0112] In addition, the sliding window-based data stream differential privacy histogram publishing method disclosed in this embodiment can be stored in a computer-readable storage medium when it is implemented in the form of a computer program as a software functional unit and sold or used as an independent product. Based on this understanding, all or part of the processes in the above-mentioned embodiment method can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps and results of the above-mentioned method embodiment. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; the storage medium can also include a combination of the above-mentioned types of memories.
[0113] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.
Claims
1. An adaptive histogram publishing method for data stream sliding window, It is characterized in that The steps are: Step 1: Determine the interval of the noise histogram to be published; Step 2: Divide the overall privacy budget ε into two parts. The first part of the privacy budget is αε, which is used to add noise to the data before sorting. The second part of the privacy budget is ε(1-α), which is used to add noise to the histogram after sorting using a greedy algorithm. Step 3: For each interval in the published histogram, use the sliding window approximate statistics to predict the estimated information of the data stream histogram at the next moment, and compare the noise value of the first part of the privacy budget with the approximate statistical value, select a suitable noise value, cluster the sorted histogram intervals and optimize the error of the data within the group, and construct the approximate statistical histogram at the current moment; The calculation process of using sliding window approximate statistics to predict the data stream histogram estimation information at the next moment is: 1) Determine whether the current data entering the sliding window data stream meets the determined statistical interval; if the current data meets the current interval range, the current data is set to 1; if the current data does not meet the current interval range, the current data is set to 0; 2) Check the position where the current data enters the sub-block: if the current data is not at the end of the sub-block, the block offset m of the current sub-block is increased by 1, and the value of the counter y is increased by 1; if the current data is at the end of the sub-block, the next data enters the next sub-block, and the sub-block offset m is reset to 0. When the value of the counter y increases by 1 and exceeds the size of the sub-block, the size of the sub-block is subtracted from the value of the counter y and the count is restarted. At the same time, the number of bits in the array b corresponding to the number of times the value of the counter y exceeds the size of the sub-block is set to 1. 3) Output the approximate statistical value at the current time i The size of the sliding window is W, the sliding window is divided into k sub-blocks, the size of the sub-block is W / k, the array of k bits is b, B is the statistic whose median of array b is 1, m*b i is the expired data of the current sliding window, is the maximum error between the approximate statistical frequency of the interval and the true frequency; 4) Judgment All results less than 0 are set to 0 to get the latest Step 4: For the histogram data selected at the current moment, add the noise value of the second part of the privacy budget to the data through the error lower limit function; Step 5: Publish the noisy histogram of the sliding window at the current moment.
2. According to claim 1, an adaptive histogram publishing method for data stream sliding window, It is characterized in that In step 2, a small sample is used for preprocessing. The optimal privacy budget allocation factor α is obtained by adjusting different privacy budget allocation factors α. The difference between the noisy histogram obtained at the current moment in the small sample and the true value at the current moment is used to obtain the optimal privacy budget allocation factor α.
3. According to the method for publishing an adaptive histogram for a data stream sliding window according to claim 2, It is characterized in that Step 3: Compare the privacy budget plus noise value of the first part with the approximate statistical value. The calculation process of selecting the appropriate noise value is as follows: 1) Calculate the approximate statistical value of the current sliding window data through approximate statistics 2) Add noise using the difference between the approximation error and the noise threshold; 3) Repeat the above steps for all intervals at the current time to obtain or In the judgment interval or All results less than 0 are set to 0 to obtain the statistical values of all intervals in the current window; 4. The method for publishing an adaptive histogram for a data stream sliding window according to claim 3, It is characterized in that The specific process of adding noise using the difference between the approximate error and the noise threshold is: (a) If the approximation error is less than the noise threshold, use the approximation Replace the noise value (b) If the noise threshold is smaller than the approximation error, use the noise value Replace approximation Among them, the approximate error is the approximate statistical result With the true value H i The difference between the noise threshold and the noise value With the true value H i The maximum difference between .
5. The method for publishing an adaptive histogram for a data stream sliding window according to claim 4, It is characterized in that Step 3 clusters the sorted histogram intervals and optimizes the error of the data within the group. The calculation process of constructing the approximate statistical histogram at the current moment is: 1) The value selected in the current sliding window Insert into current histogram statistics; 2) First, use the sorting algorithm to sort the statistical data in the sliding window; and calculate the error between each frequency and other frequencies when combined and the error when not combined; 3) Determine the interval grouping; a) When the error of merging two intervals is less than the error of not merging, the two intervals are merged and set as one group; b) When the error of merging two intervals is greater than the error of not merging, the two intervals are not merged and are set as two groups; Among them, the merged error is the result of merging the intervals and taking the square difference between the average value of each interval and the existing value of each interval; the unmerged error is the result of not merging the intervals and taking the square difference between the average value of each interval and the existing value of each interval; 4) Group the data in the current sliding window using the above interval grouping method; 5) Use the average value of each group to replace the frequency value in each group, and determine the privacy budget in each interval by the proportion of the statistical value in each group to the total number of data statistical values in the entire sliding window to add Laplace noise; 6) In order to reduce the error caused by approximate statistics, judge All results less than 0 are set to 0, and the values that do not meet the conditions are filtered out. Finally, the noise histogram at the current time i is returned.
6. The method for publishing an adaptive histogram for a data stream sliding window according to claim 5, It is characterized in that Step 4: The process of obtaining the noised histogram of the second privacy budget by adding the error lower limit function to the current histogram data: The first interval of the sorted approximate statistical histogram forms a group by itself, and the noise error introduced by the second interval after being added to the current group and the noise error introduced by the new group re-established with the second interval as the starting point are judged; if the noise error introduced by adding the current group is smaller than that introduced by establishing a new group, then add the current group, otherwise establish a new group; according to this rule, add noise to all histograms.
7. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the method for publishing an adaptive histogram for a data stream sliding window as described in any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Histogram-based data flow-oriented differential privacy publishing method
CN105046160A
Data flow differential privacy histogram publishing method based on sliding window
CN112307078A
