An online streaming sampling publication method and system with differential privacy

Abstract

The application provides an online streaming sampling publishing method and system with differential privacy, and relates to the field of information security; through one-time scanning of a data stream, attribute statistical information of each element in the data stream is stored into a data structure SES with high processing capacity, and then histogram generation and publishing are performed according to the collected data of the data structure SES; the published data has the same privacy protection strength guarantee as the differential privacy algorithm; wherein the data structure SES approximately stores each sliding window, which significantly reduces the storage cost, and according to the collected data, the histogram can be quickly generated, thereby reducing the running time in the data processing process.

Description

Technical Field

[0001] This invention relates to the field of information security technology, specifically to an online streaming sampling and publishing method and system with differential privacy. Background Technology

[0002] With the widespread adoption of 5G networks and mobile devices, massive amounts of private data are being generated at high speed, forming an endless data stream. The application of these data streams can provide users with better services. For example, when data streams are used in network traffic analysis, websites collect information from users' network traffic daily, including personal browsing information (click count, search count, etc.). Website staff can analyze this browsing information to recommend items based on user preferences. When data streams are used in vehicle traffic monitoring, vehicle monitoring instruments collect vehicle location and movement data daily. In real-time traffic information systems, the system analyzes the current traffic conditions in each area based on information uploaded by vehicles, helping vehicles plan their journeys to their destinations more quickly and predicting future traffic conditions, effectively avoiding losses caused by traffic congestion.

[0003] Based on the above practical applications, it is clear that with the rapid development of information technology, many existing applications often need to dynamically publish statistical data on the information they generate or obtain. However, this publishing process may lead to the leakage of users' privacy data. How to securely and consciously publish histograms of the latest data items (sliding windows) in such streams has become a fundamental issue, as it not only allows the public to understand current collective phenomena (such as flu outbreaks and traffic congestion) but also prevents the leakage of personal privacy.

[0004] Research has found that existing practical applications, including the aforementioned examples of network traffic analysis and vehicle traffic monitoring, need to place greater emphasis on recent data rather than historical data. Recent data better indicates trends in current conditions and requires rapid processing. For instance, Apple retains data collected from users for a maximum of three months; from a recommendation perspective, any data exceeding this period is likely to be considered outdated. While Apple limits the leakage of personal data to some extent by retaining only the most recent user data, it can still lead to the leakage of personal privacy data. Therefore, it is necessary to study the privacy issues of data publication on sliding windows, propose how to perform rapid data processing on sliding window models, and how to implement privacy-preserving data publication.

[0005] Existing methods for histogram publishing based on differential privacy are not suitable for histogram publishing in data stream sliding window models, and their application has the following drawbacks:

[0006] (1) Existing data flow methods rarely focus on the latest elements and do not have a method for quickly calculating data in the sliding window; (2) Existing methods need to scan the data in each sliding window while constructing the histogram, which leads to high runtime and storage overhead; (3) Existing methods do not have a method for quickly constructing the histogram and need to perform an overall count of the data before publishing, which greatly delays the publishing speed. Summary of the Invention

[0007] The purpose of this invention is to provide an online streaming sampling and publishing method and system with differential privacy, which solves the problems of existing methods applied to the histogram publishing process of data stream sliding window models, which rarely pay attention to the latest elements, require scanning the data of each sliding window at the same time, resulting in high running and storage overhead and low publishing speed. By scanning the data stream once, the method stores the attribute statistics of each element in the data stream into a data structure SES with high processing capabilities, and then publishes the histogram based on the collected data of the data structure SES. The published data has the same level of privacy protection as the differential privacy algorithm.

[0008] To achieve the above objectives, the present invention proposes the following technical solution: an online streaming sampling and publishing method with differential privacy, comprising the following steps:

[0009] Determine the intervals of the histogram of data to be published in the data stream;

[0010] For all intervals of the published histogram, the data within the sliding window at the current time is sampled using the data structure SES to obtain the sample set at the current time.

[0011] Based on the current sampling set, obtain the statistical results of all intervals within the current sliding window;

[0012] Based on the statistical results, publish the histogram of the sliding window at the current moment;

[0013] The process by which the data structure SES samples the data within the sliding window at the current moment is as follows:

[0014] Define the size of the sliding window as w, and the data at the current time t as e. t The current data stream contains an odd-numbered set of samples within an odd-numbered sliding window. S 1. The set of even-numbered samples of data within the even-numbered sliding window of the data stream at the current moment. S 2. Sampling set S The set S 1. S 2 and S The size of each is s, and w > s;

[0015] When the data stream enters the first sliding window, and the data length is 1 to w, local reservoir sampling is used to obtain an odd-numbered sampling set. S 1, and the sampling set S Equal to odd sample set S 1;

[0016] When the data stream enters the second sliding window, and the data length is w+1 to 2w, a local reservoir sampling method is used to obtain an even-numbered sampling set. S 2; the sampling set S The structure is as follows: at any given time, determine the odd-numbered sample set. S 1. Does any element expire? Then, delete the odd-numbered sample set of expired elements. S 1 and even sample sets S 2. Merge to generate a sample set S When the data stream enters the second sliding window at time t=2w, the odd-numbered sampling set... S All elements with a value of 1 are expired; the current sample set is... S Equal to even sample set S 2;

[0017] When the data stream enters the sliding window at time t > 2w, the sliding window into which the data stream flows is divided into odd-numbered windows (mod(fix(t / w), 2) = 0) and even-numbered windows (mod(fix(t / w), 2) = 1). Local reservoir sampling is used for any odd-numbered or even-numbered window to obtain the sample set at each time step. S ;

[0018] The process of local reservoir sampling is as follows: for data in the data stream located 1-2 seconds before the sliding window, the data at the current time e is directly... t Place it into the corresponding sampling set; for the data in the data stream located at positions s~w after the sliding window, add a random factor r, r∈[1,t], to any data. When r≤s, replace the element in the sampling set corresponding to the position of the random factor r with the data at the current moment in the data stream; otherwise, keep the data in the sampling set unchanged.

[0019] Furthermore, the sample set obtained from the data stream at any given time... S It includes the element of the current time of the data stream, the sampling statistics of all intervals, and the timestamp t;

[0020] For 1~ w Sample set for each time step within the time range S Statistical counting was used to obtain the sampling statistics for all intervals.

[0021] for w+1~2 w Time range, odd-numbered windows at times t > 2w ( mod ( fix ( t / w ), 2) = 0) and even windows ( mod ( fix ( t / w The sample set for each time step within the range of ), 2) = 1). S Based on the sampling statistics of all intervals at the previous time and whether there are expired elements in the data at the current time, the statistical results of each interval at the current time are corrected. Only one element in the sliding window at any time is expired.

[0022] Furthermore, the process of obtaining the statistical results of all intervals within the sliding window at the current moment is as follows:

[0023] The sample set at the current moment S The data in the obtained sample set are summed using the basic counting method to obtain the sample set at the current time. S The statistical results of all intervals within the range are denoted as { count1, count2, count3,…, countI},in, I For all possible intervals of the histogram;

[0024] Based on the random selection property, estimate the statistical results of all intervals within the sliding window at the current time, denoted as { count1, count2, count3,…, countI}*( w / s ).

[0025] Furthermore, the histogram of the sliding window at the current moment published satisfies (ε,δ)-differential privacy.

[0026] Another technical solution of the present invention discloses an online streaming sampling and publishing system with differential privacy, the system comprising:

[0027] The determination module is used to determine the intervals of the histogram of data to be published in the data stream;

[0028] The sampling module is used to sample all intervals of the histogram to be published. It uses the data structure SES to sample the data within the sliding window at the current time and obtain the sample set at the current time.

[0029] The acquisition module is used to obtain the statistical results of all intervals within the sliding window at the current time, based on the sampling set at the current time.

[0030] The publishing module is used to publish the histogram of the sliding window at the current moment based on the statistical results.

[0031] The sampling module, which implements the SES data structure to sample the data within the sliding window at the current moment, includes the following execution units:

[0032] Define a unit to define the size of the sliding window as w and the data at the current time t of the data stream as e. t The current data stream contains an odd-numbered set of samples within an odd-numbered sliding window. S 1. The set of even-numbered samples of data within the even-numbered sliding window of the data stream at the current moment. S 2. Sampling set S The set S 1. S The size of both 2 and S is s, and w > s;

[0033] The first sampling unit is used to obtain an odd-numbered sampling set by using local reservoir sampling when the data stream enters the first sliding window and the data length is 1 to w. S 1, and the sampling set S Equal to odd sample set S 1;

[0034] The second sampling unit is used to obtain an even-numbered sampling set by using local reservoir sampling when the data stream enters the second sliding window and the data length is w+1 to 2w. S 2; the sampling set S The structure is as follows: at any given time, determine the odd-numbered sample set. S 1. Does any element expire? Then, delete the odd-numbered sample set of expired elements. S 1 and even sample sets S 2. Merge to generate a sample set S When the data stream enters the second sliding window at time t=2w, the odd-numbered sampling set... S All elements with a value of 1 are expired; the current sample set is... S Equal to even sample set S 2;

[0035] The third sampling unit is used to divide the sliding window into odd-numbered windows (mod(fix(t / w), 2) = 0) and even-numbered windows (mod(fix(t / w), 2) = 1) when the data stream enters the sliding window at the current time t > 2w. Local reservoir sampling is then used for any odd-numbered or even-numbered window to obtain the sample set at each time step. S ;

[0036] The process of local reservoir sampling is as follows: for data in the data stream located 1 to 2 seconds before the sliding window, the data at the current time e is directly obtained. tPlace it into the corresponding sampling set; for the data in the data stream located at positions s~w after the sliding window, add a random factor r, r∈[1,t], to any data. When r≤s, replace the element in the sampling set corresponding to the position of the random factor r with the data at the current moment in the data stream; otherwise, keep the data in the sampling set unchanged.

[0037] Furthermore, the sampling set obtained by the sampling module at the current moment includes the element of the current time of the data stream, the sampling statistics results of all intervals, and the timestamp t. The process of obtaining the sampling statistics results of all intervals is as follows:

[0038] For 1~ w Sample set for each time point within the time range S Statistical counting was used to obtain the sampling statistics for all intervals.

[0039] for w +1~2 w Time range, odd-numbered windows at time t > 2w ( mod ( fix ( t / w ), 2) = 0) and even windows ( mod ( fix ( t / w The sample set for each time step within the range of ), 2) = 1). S Based on the sampling statistics of all intervals at the previous time and whether there are expired elements in the data at the current time, the statistical results of each interval at the current time are corrected. Only one element in the sliding window at any time is expired.

[0040] Furthermore, the execution unit of the acquisition module, which retrieves the statistical results of all intervals within the sliding window at the current time, includes:

[0041] The first computational unit is used for the sample set at the current moment. S The data in the obtained sample set are summed using the basic counting method to obtain the sample set at the current time. S The statistical results of all intervals within the range are denoted as { count1, count2, count3,…, countI},in, I For all possible intervals of the histogram;

[0042] The second calculation unit is used to estimate the statistical results of all intervals within the sliding window at the current time, based on the properties of random selection, denoted as { count1, count2, count3,…, countI}*( w / s ).

[0043] Furthermore, the histogram of the sliding window at the current moment published by the publishing module satisfies (ε,δ)-differential privacy.

[0044] The present invention also provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the above-described online streaming sampling and publishing method with differential privacy.

[0045] Another technical solution of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the online streaming sampling and publishing method with differential privacy as described above.

[0046] As can be seen from the above technical solutions, the technical solutions of the present invention have achieved the following beneficial effects:

[0047] Compared with existing technologies, the online streaming sampling and publishing method and system with differential privacy disclosed in this invention adopts a sliding window model to observe the latest elements generated in the data stream under big data environment in order to reduce the data publishing processing time and space overhead per timestamp. Furthermore, to further reduce the data publishing processing time and space overhead per timestamp, a novel data structure SES is proposed to quickly process the data within each sliding window. The SES data structure can continuously process these data results and quickly obtain the sample set of data within the sliding window at each time point without caching all the data within the sliding window at the current time point, thus greatly reducing storage costs and approximately storing each sliding window. The SES data structure is applicable to the sliding window model and can be used for continuous histogram publishing, thus estimating the count value of the current sliding window with lower space overhead.

[0048] This invention further implements a method for rapid histogram generation based on the SES data structure. Specifically, it utilizes the data from the previous time step and obtains the statistical count value at the current time step based on the properties of the sliding window model and the correlation of the sample set. This significantly improves the efficiency of data result query and data counting, and provides the same privacy protection as differential privacy.

[0049] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other.

[0050] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0051] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein:

[0052] Figure 1 This is a flowchart of the online streaming sampling and publishing method with differential privacy according to the present invention;

[0053] Figure 2 This is an example diagram of the sampling set of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art.

[0055] The terms "first," "second," and similar words used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "an," "a," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. Terms such as "comprising" or "including" indicate that the element or object preceding "comprising" encompasses the features, integrals, steps, operations, elements, and / or components listed following "comprising" or "including," and do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0056] While existing technologies offer various sliding window histogram-based publishing methods for data stream environments, they lack a fast way to compute data within the sliding window. These methods require scanning data in each sliding window while constructing the histogram, leading to high runtime and storage overhead. Therefore, this invention provides an online streaming sampling and publishing method and system with differential privacy. Addressing the characteristics of large volumes, high speed, and real-time processing in data streams, and the need to focus on the latest elements, a data structure called SES is proposed. This structure models the data stream using a series of sliding windows. Based on the sampled data from the previous time step, the properties of the sliding window model, and the correlation of the sample set, the statistical count value for the current time step is obtained, and then a histogram is published. The data generated by this method achieves the same level of privacy protection as differential privacy with added noise.

[0057] The following detailed description, in conjunction with specific embodiments, further illustrates the online streaming sampling and publishing method and system with differential privacy disclosed in this invention.

[0058] The online streaming sampling and publishing method with differential privacy disclosed in the embodiment, denoted as the PAS algorithm, includes the following steps:

[0059] Step S102: Determine the interval of the histogram of the data to be published in the data stream;

[0060] Step S104: For all intervals of the histogram to be published, use the data structure SES to sample the data within the sliding window at the current time to obtain the sample set at the current time.

[0061] Specifically: Define the size of the sliding window as w, and the current time of the data stream... t The data is e t The current data stream contains an odd-numbered set of samples within an odd-numbered sliding window. S 1. The set of even-numbered samples of data within the even-numbered sliding window of the data stream at the current moment. S 2. Sampling set S The set S 1. S 2 and S The size of each is s, and w > s;

[0062] When the data stream enters the first sliding window, and the data length is 1 to w, local reservoir sampling is used to obtain an odd-numbered sampling set. S 1, and the sampling set S Equal to odd sample set S 1;

[0063] When the data stream enters the second sliding window, and the data length is w+1 to 2w, a local reservoir sampling method is used to obtain an even-numbered sampling set.S 2; the sampling set S The structure is as follows: at any given time, determine the odd-numbered sample set. S 1. Does any element expire? Then, delete the odd-numbered sample set of expired elements. S 1 and even sample sets S 2. Merge to generate a sample set S When the data stream enters the second sliding window at time t=2w, the odd-numbered sampling set... S All elements with a value of 1 are expired; the current sample set is... S Equal to even sample set S 2;

[0064] When the data stream enters the sliding window at time t > 2w, the sliding window into which the data stream flows is divided into odd-numbered windows (mod(fix(t / w), 2) = 0) and even-numbered windows (mod(fix(t / w), 2) = 1). Local reservoir sampling is used for any odd-numbered or even-numbered window to obtain the sample set at each time step. S When sampling in odd-numbered windows (e.g., (2w+1~3w, 4w+1~5w, 6w+1~7w, ...) and even-numbered windows (e.g., (3w+1~4w, 5w+1~6w, 7w+1~8w, ...), the data is first converted to the range 1~w, and then odd-numbered sampling sets are obtained based on local reservoir sampling. S 1 and even sample sets S 2.

[0065] The process of local reservoir sampling is as follows: for data in the data stream located 1 to 2 seconds before the sliding window, the data at the current time e is directly obtained. t Place the data into the corresponding sampling set; for data in the data stream located at positions s~w after the sliding window, add a random factor r, r∈[1, t], to any data. When r≤s, replace the element in the sampling set corresponding to the position of the random factor r with the data of the current time in the data stream; otherwise, keep the data in the sampling set unchanged. That is, if the current time t belongs to the data between positions 1~s in the sliding window, we directly add the current time data e t Place it into the odd or even sampling set; then, when t belongs to the data between s and w, for the odd or even sampling set, the data that flowed in first has already added all the positions in the set. Using a random factor r, for the data at the current time, e is added with probability s / t. t Insert into the odd or even sample set, with a probability of 1-s / t, e t Do not insert into odd or even sample sets.

[0066] Step S106: Based on the sampling set at the current time, obtain the statistical results of all intervals within the sliding window at the current time;

[0067] This step essentially implements the process of quickly generating a histogram based on the data structure SES, given the sample set obtained from the data stream DS at any time in step S104. S It includes the elements of the data stream at the current time, the sampling statistics of all intervals, and the timestamp t; the process of obtaining the sampling statistics of all intervals in the sampling set at the current time is as follows:

[0068] For 1~ w Sample set for each time step within the time range S Statistical counting was used to obtain the sampling statistics for all intervals.

[0069] for w +1~2 w Time range, odd-numbered windows at time t > 2w ( mod ( fix ( t / w ), 2) = 0), such as the interval (2 w +1~3 w 4 w +1~5 w (,...) and even windows ( mod ( fix ( t / w ), 2) = 1), such as the sample set for each time step within the interval (3w+1~4w, 5w+1~6w,…). S The statistical results of each interval at the current time are corrected based on the sampling statistics of all intervals at the previous time and whether there are expired elements in the data at the current time. Only one element in the sliding window at any time is expired.

[0070] For example, when considering the sample set for each time step within the range of w+1 to 2w... S Based on the data results from the previous moment and the data from the current moment, i.e., the odd-numbered sampling set. S 1. Whether there are expired elements in the sample set is used to obtain the count result at the current moment, without having to perform statistics on the entire sample set.

[0071] The current sampling set S Based on the above acquisition process, the data in the obtained sample set can be summed directly using the basic counting method to obtain the sample set at the current time. S The statistical results of all intervals within the range are denoted as { count1, count2, count3,…, countI},in, IFor all possible intervals of the histogram;

[0072] Based on the random selection property, estimate the statistical results of all intervals within the sliding window at the current time, denoted as { count1, count2, count3,…, countI}*( w / s That is, the count estimate of all intervals of the sliding window is based on the statistical results of the sample set multiplied by the multiple between the current sliding window and the sample set.

[0073] Step S108: Based on the statistical results, publish the histogram of the sliding window at the current moment;

[0074] The above-disclosed online streaming sampling and publishing method with differential privacy performs a single-pass scan of a given data stream DS and uses a sliding window model for statistical publishing. The core of the method includes two points: first, it proposes a novel data structure SES, namely a sampling method based on a sliding window model, which can store the key statistical information of each element in DS into an efficient memory-based SES data structure; second, it proposes a fast histogram generation method based on the SES data structure. This method can utilize the sampled data from the previous time step of the data stream, as well as the properties of the sliding window model and the correlation of the sample sets, to obtain the statistical count value at the current time step and perform histogram publishing; and the histogram of the current time step sliding window satisfies (ε,δ)-differential privacy.

[0075] Proof: In existing research, the sliding window histogram publishing method generally sets δ=10⁻⁵. Based on the fact that the error generated by the Gaussian distribution is equal to the error generated by the PAS algorithm, we obtain:

[0076] ;

[0077] Substitute δ=10 -5 The following equation is obtained:

[0078] ;

[0079] in, C This indicates that the current sliding window belongs to the interval. I The specific number of elements;

[0080] The noise error of all intervals of the histogram is analyzed as follows, defining a function. f ( C ):

[0081] ;

[0082] based on C This indicates that the current sliding window belongs to the interval. IThe specific number of elements, for different intervals, C They have different values;

[0083] Assuming there is in the data stream I Intervals, and through C 1. C 2、…、C I express I The actual counts of each interval are then: f ( C 1), f ( C 2), f ( C I ) represent the privacy budget allocated to each interval {ε1, ε2, ..., ε I Research on Parallel Combinatorial Principles Based on Differential Privacy {ε1, ε2, ..., ε I The maximum value of the differential privacy parameter across multiple intervals, and the privacy budget that satisfies the privacy protection requirements of all intervals as long as one of the intervals has the maximum privacy budget, i.e., (ε,δ)-differential privacy.

[0084] Time complexity analysis

[0085] Given odd sample set S 1 and even sample sets S If the size of 2 is s, then the time cost of the PAS algorithm is O( ). s ).

[0086] Proof: For any sliding window, the processing of the PAS algorithm mainly consists of: (1) a sampling method based on the sliding window model; (2) a fast histogram generation method based on the SES data structure; among which, the main time cost is to obtain an odd sample set through local reservoir sampling. S 1 and even sample sets S The processing time required is 2, therefore the time complexity of the PAS algorithm is O(2). s ).

[0087] Assumption l This represents the number of all possible intervals in the published histogram. w The size of the sliding window. s Given the size of the sampling set, the average space cost of the current PAS algorithm is at most ( log 2 l + log 2( w / s )+ log 2 log 2( w -s +1))(1.5 s - s 2 / 2 w ).

[0088] Proof: Without loss of generality, the study of the data structure SES in { w +1, w +2, ..., w+w}, w The spatial cost in the environment was determined, and the average spatial cost expenditure was found;

[0089] Given the current time t∈ { t w+1 , t w+2 , ..., t w+k}, t When the integer is positive, the space cost of SES analysis of the data structure is the same; let g ∈ {1, 2, ..., w}, g For time, then:

[0090] (1) In w+g The moment ( g ∈{1,2,…, w Odd-numbered sample sets S The unexpired sample elements in 1 are s ×(1- g / w );

[0091] (2) At time w+g (g∈{1,2,…,g}(g≤) s Even-numbered sample sets S The sample size in 2 is g, and the sample set is even. S None of the samples in 2 have expired; only odd-numbered sample sets exist. S Elements that have not expired in section 1 are retained;

[0092] (3) At time w+g (g∈{s+1, s+2, ..., w}), g Randomly selected from the samples s There are 10 data points, so the even-numbered sample set S The number of samples in 2 is s ;

[0093] As shown in (1)-(3) above:

[0094] a) At time w+g (g∈{1,2,…,s}), the space cost required by the PAS algorithm is [s(1-g / w)+g]×[log 2 l + log 2(ws)+ log 2 log 2(w-s+1)] bits;

[0095] b) At time w+g (g∈{s+1, s+2, ..., w}), the space cost required by the PAS algorithm is [s(1-g / w)+g]×[ log 2 l + log 2(w / s)+ log 2 log 2(w-s+1)] bits;

[0096] Let M = log 2 l + log 2( w / s )+ log 2 log 2( w - s +1), then the average number of bits required for the space overhead of the PAS algorithm is M / w(1.5ws -s). 2 / 2) = M(1.5ss 2 / 2 w ).

[0097] The online streaming sampling and publishing method with differential privacy disclosed in this embodiment can improve the generation speed of query histogram data and enhance the query efficiency of data structures, while rapidly processing data streams within a sliding window.

[0098] In an embodiment of the present invention, an electronic device is also provided, the device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described online streaming sampling and publishing method with differential privacy.

[0099] The aforementioned program can run in a processor or be stored in memory (or a computer-readable storage medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves.

[0100] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented using different modules, and different steps can be implemented using different modules.

[0101] In this embodiment, a system is provided, which can be called an online streaming sampling and publishing system with differential privacy, the system comprising:

[0102] The determination module is used to determine the intervals of the histogram of data to be published in the data stream;

[0103] The sampling module is used to sample all intervals of the histogram to be published. It uses the data structure SES to sample the data within the sliding window at the current time and obtain the sample set at the current time.

[0104] The acquisition module is used to obtain the statistical results of all intervals within the sliding window at the current time, based on the sampling set at the current time.

[0105] The publishing module is used to publish the histogram of the sliding window at the current time based on statistical results. This histogram has the same differential privacy strength as the histogram generated based on differential privacy.

[0106] The sampling module, which implements the SES data structure to sample the data within the sliding window at the current moment, includes the following execution units:

[0107] Define a unit to define the size of the sliding window as w and the data at the current time t of the data stream as e. t The current data stream contains an odd-numbered set of samples within an odd-numbered sliding window. S 1. The set of even-numbered samples of data within the even-numbered sliding window of the data stream at the current moment. S 2. Sampling set S The set S 1. S 2 and S The size of each is s, and w > s;

[0108] The first sampling unit is used to obtain an odd-numbered sampling set by using local reservoir sampling when the data stream enters the first sliding window and the data length is 1 to w. S 1, and the sampling set S Equal to odd sample set S 1;

[0109] The second sampling unit is used to obtain an even-numbered sampling set by using local reservoir sampling when the data stream enters the second sliding window and the data length is w+1 to 2w. S 2; the sampling set S The structure is as follows: at any given time, determine the odd-numbered sample set. S 1. Does any element expire? Then, delete the odd-numbered sample set of expired elements. S 1 and even sample sets S 2. Merge to generate a sample set S When the data stream enters the second sliding window at time t=2w, the odd-numbered sampling set... S All elements with a value of 1 are expired; the current sample set is... S Equal to even sample set S 2;

[0110] The third sampling unit is used to divide the sliding window into odd-numbered windows (mod(fix(t / w), 2) = 0) and even-numbered windows (mod(fix(t / w), 2) = 1) when the data stream enters the sliding window at the current time t > 2w. Local reservoir sampling is then used for any odd-numbered or even-numbered window to obtain the sampling set at each time step. S ;

[0111] The process of local reservoir sampling is as follows: for data in the data stream located 1 to 2 seconds before the sliding window, the data at the current time e is directly obtained. tPlace it into the corresponding sampling set; for the data in the data stream located at positions s~w after the sliding window, add a random factor r, r∈[1,t], to any data. When r≤s, replace the element in the sampling set corresponding to the position of the random factor r with the data at the current moment in the data stream; otherwise, keep the data in the sampling set unchanged.

[0112] The steps of the system for implementing the online streaming sampling and publishing method in the above embodiments have already been described and will not be repeated here.

[0113] For example, the sampling set obtained by the sampling module at the current moment includes the elements of the current time of the data stream, the sampling statistics of all intervals, and the timestamp t. The process of obtaining the sampling statistics of all intervals is as follows:

[0114] For 1~ w Sample set for each time step within the time range S Statistical counting was used to obtain the sampling statistics for all intervals.

[0115] for w +1~2 w Time range, odd-numbered windows at time t > 2w ( mod ( fix ( t / w ), 2) = 0) and even windows ( mod ( fix ( t / w The sample set for each time step within the range of ), 2) = 1). S Based on the sampling statistics of all intervals at the previous time and whether there are expired elements in the data at the current time, the statistical results of each interval at the current time are corrected. Only one element in the sliding window at any time is expired.

[0116] For example, the execution unit that implements the acquisition module to obtain the statistical results of all intervals within the sliding window at the current time includes:

[0117] The first computational unit is used for the sample set at the current moment. S The data in the obtained sample set are summed using the basic counting method to obtain the sample set at the current time. S The statistical results of all intervals within the range are denoted as { count1, count2, count3,…, countI},in, I For all possible intervals of the histogram;

[0118] The second calculation unit is used to estimate the statistical results of all intervals within the sliding window at the current time, based on the properties of random selection, denoted as { count1, count2, count3,…, countI}*( w / s ).

[0119] This invention, on the one hand, uses the SES data structure to collect data from the sliding window model, which can greatly reduce storage costs to approximately store each sliding window; on the other hand, it uses a fast histogram generation method based on the SES data structure to improve the generation speed of histograms and reduce the running time in the data processing process; this invention can quickly process data streams and rapidly publish histogram data with strong differential privacy protection.

[0120] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for online streaming sampling and publishing with differential privacy, characterized in that, Includes the following steps: Determine the intervals of the histogram of data to be published in the data stream; For all intervals of the published histogram, the data within the sliding window at the current time is sampled using the data structure SES to obtain the sample set at the current time. Based on the current sampling set, obtain the statistical results of all intervals within the current sliding window; Based on the statistical results, publish the histogram of the sliding window at the current moment; The process by which the data structure SES samples the data within the sliding window at the current moment is as follows: Define the size of the sliding window as w, and the data at the current time t as e. t The current data stream contains an odd-numbered set of samples within an odd-numbered sliding window. S 1. The set of even-numbered samples of data within the even-numbered sliding window of the data stream at the current moment. S 2. Sampling set S The set S 1. S 2 and S The size of each is s, and w > s; When the data stream enters the first sliding window, and the data length is 1 to w, local reservoir sampling is used to obtain an odd-numbered sampling set. S 1, and the sampling set S Equal to odd sample set S 1; When the data stream enters the second sliding window, and the data length is w+1 to 2w, a local reservoir sampling method is used to obtain an even-numbered sampling set. S 2; the sampling set S The structure is as follows: at any given time, determine the odd-numbered sample set. S 1. Does any element expire? Then, delete the odd-numbered sample set of expired elements. S 1 and even sample sets S 2. Merge to generate a sample set S When the data stream enters the second sliding window at time t=2w, the odd-numbered sampling set... S All elements with a value of 1 are expired; the current sample set is... S Equal to even sample set S 2; When the data stream enters the sliding window at time t > 2w, the sliding window into which the data stream flows is divided into odd-numbered windows (mod(fix(t / w), 2) = 0) and even-numbered windows (mod(fix(t / w), 2) = 1). Local reservoir sampling is used for any odd-numbered or even-numbered window to obtain the sample set at each time step. S ; The process of local reservoir sampling is as follows: for data in the data stream located 1-2 seconds before the sliding window, the data at the current time e is directly... t Place it into the corresponding sampling set; for the data in the data stream located at positions s~w after the sliding window, add a random factor r, r∈[1,t], to any data. When r≤s, replace the element in the sampling set corresponding to the position of the random factor r with the data at the current moment in the data stream; otherwise, keep the data in the sampling set unchanged.

2. The online streaming sampling and publishing method with differential privacy according to claim 1, characterized in that, The sample set obtained for the data stream at any given time S It includes the element of the current time of the data stream, the sampling statistics of all intervals, and the timestamp t; For 1~ w Sample set for each time point within the time range S Statistical counting was used to obtain the sampling statistics for all intervals. for w +1~2 w Time range, odd-numbered windows at times t > 2w ( mod ( fix ( t / w ), 2) = 0) and even windows ( mod ( fix ( t / w The sample set for each time step within the range of ), 2) = 1). S Based on the sampling statistics of all intervals at the previous time and whether there are expired elements in the data at the current time, the statistical results of each interval at the current time are corrected. Only one element in the sliding window at any time is expired.

3. The online streaming sampling and publishing method with differential privacy according to claim 2, characterized in that, The process of obtaining the statistical results of all intervals within the sliding window at the current time is as follows: The sample set at the current moment S The data in the obtained sample set are summed using the basic counting method to obtain the sample set at the current time. S The statistical results of all intervals within the range are denoted as { count1, count2, count3,…, countI },in, I For all possible intervals of the histogram; Based on the random selection property, estimate the statistical results of all intervals within the sliding window at the current time, denoted as { count1, count2, count3, ..., countI }*( w / s ).

4. The online streaming sampling and publishing method with differential privacy according to claim 1, characterized in that, The histogram of the sliding window at the current moment published satisfies (ε,δ)-differential privacy.

5. An online streaming sampling and publishing system with differential privacy, characterized in that, include: The determination module is used to determine the intervals of the histogram of data to be published in the data stream; The sampling module is used to sample all intervals of the histogram to be published. It uses the data structure SES to sample the data within the sliding window at the current time and obtain the sample set at the current time. The acquisition module is used to obtain the statistical results of all intervals within the sliding window at the current time, based on the sampling set at the current time. The publishing module is used to publish the histogram of the sliding window at the current moment based on the statistical results. The sampling module, which implements the SES data structure to sample the data within the sliding window at the current moment, includes the following execution units: Define a unit to define the size of the sliding window as w and the data at the current time t of the data stream as e. t The current data stream contains an odd-numbered set of samples within an odd-numbered sliding window. S 1. The set of even-numbered samples of data within the even-numbered sliding window of the data stream at the current moment. S 2. Sampling set S The set S 1. S 2 and S The size of each is s, and w > s; The first sampling unit is used to obtain an odd-numbered sampling set by using local reservoir sampling when the data stream enters the first sliding window and the data length is 1 to w. S 1, and the sampling set S Equal to odd sample set S 1; The second sampling unit is used to obtain an even-numbered sampling set by using local reservoir sampling when the data stream enters the second sliding window and the data length is w+1 to 2w. S 2; the sampling set S The structure is as follows: at any given time, determine the odd-numbered sample set. S 1. Does any element expire? Then, delete the odd-numbered sample set of expired elements. S 1 and even sample sets S 2. Merge to generate a sample set S When the data stream enters the second sliding window at time t=2w, the odd-numbered sampling set... S All elements with a value of 1 are expired; the current sample set is... S Equal to even sample set S 2; The third sampling unit is used to divide the sliding window into odd-numbered windows (mod(fix(t / w), 2) = 0) and even-numbered windows (mod(fix(t / w), 2) = 1) when the data stream enters the sliding window at the current time t > 2w. Local reservoir sampling is then used for any odd-numbered or even-numbered window to obtain the sampling set at each time step. S ; The process of local reservoir sampling is as follows: for data in the data stream located 1 to 2 seconds before the sliding window, the data at the current time e is directly input. t Place it into the corresponding sampling set; for the data in the data stream located at positions s~w after the sliding window, add a random factor r, r∈[1,t], to any data. When r≤s, replace the element in the sampling set corresponding to the position of the random factor r with the data at the current moment in the data stream; otherwise, keep the data in the sampling set unchanged.

6. The online streaming sampling and publishing system with differential privacy according to claim 5, characterized in that, The sampling set obtained by the sampling module at the current moment includes the element of the current time of the data stream, the sampling statistics results of all intervals, and the timestamp t. The process of obtaining the sampling statistics results of all intervals is as follows: For 1~ w Sample set for each time step within the time range S Statistical counting was used to obtain the sampling statistics for all intervals. for w +1~2 w Time range, odd-numbered windows at times t > 2w ( mod ( fix ( t / w ), 2) = 0) and even windows ( mod ( fix ( t / w The sample set for each time step within the range of ), 2) = 1). S Based on the sampling statistics of all intervals at the previous time and whether there are expired elements in the data at the current time, the statistical results of each interval at the current time are corrected. Only one element in the sliding window at any time is expired.

7. The online streaming sampling and publishing system with differential privacy according to claim 6, characterized in that, The acquisition module, which implements the execution unit for obtaining the statistical results of all intervals within the sliding window at the current time, includes: The first computational unit is used for the sample set at the current moment. S The data in the obtained sample set are summed using the basic counting method to obtain the sample set at the current time. S The statistical results of all intervals within the range are denoted as { count1, count2, count3,…, countI },in, I For all possible intervals of the histogram; The second calculation unit is used to estimate the statistical results of all intervals within the sliding window at the current time, based on the properties of random selection, denoted as { count1, count2, count3,…, countI }*( w / s ).

8. The online streaming sampling and publishing system with differential privacy according to claim 5, characterized in that, The histogram of the sliding window published by the publishing module at the current moment satisfies (ε,δ)-differential privacy.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the online streaming sampling and publishing method with differential privacy as described in any one of claims 1-4.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the online streaming sampling and publishing method with differential privacy as described in any one of claims 1-4.

Images