Prefix and determination method and device, electronic equipment and storage medium

Abstract

The invention provides a prefix and determination method and device, electronic equipment and a storage medium. The method comprises the following steps: converting a timestamp of a current data element in a data stream into a binary number; and updating a cardinal number array based on digital elements corresponding to the data elements in a digital stream and the binary number, and updating a noise array based on the binary number, the digital elements in the digital stream being in one-to-one correspondence with the data elements in the data stream, and the data elements in the data stream being in one-to-one correspondence with the data elements in the data stream. Each element of the cardinal number array is used for representing the cardinal number of a layer of nodes of the binary tree, and each element of the noise array is used for representing the noise of the layer of nodes of the binary tree; and determining a prefix sum of the digital stream based on the binary number, the cardinal number array and the noise array, the prefix sum of the digital stream being used for determining a cardinal number estimation value of the data stream.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of database technology, and in particular, to a method and apparatus for determining prefix sum, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of Internet technology and big data, institutions and enterprises can obtain a vast amount of user data. By deeply analyzing this data, many valuable pieces of information can be extracted to help relevant departments, enterprises, and users make key decisions. For example, social platforms can accurately recommend content that users may be interested in based on their browsing records and like behaviors; e-commerce platforms analyze users' search, click, and favorite data to understand their preferences and predict shopping trends; video websites use the view count data of videos to evaluate the popularity of videos.

[0003] However, large-scale data collection and analysis also bring risks of privacy leakage. Potential attackers may reverse-engineer users' personal information from the publicly available analysis results, thus causing adverse effects on users. Therefore, in the process of data analysis, how to effectively protect user privacy has become an important issue that urgently needs to be solved. Currently, differential privacy is widely regarded as the most general privacy protection standard in statistical data analysis.

[0004] Many application scenarios require real-time cardinality statistics monitoring of a large amount of data streams. For example, for the video viewing data of video websites, cardinality estimation needs to be performed for each video. However, when performing cardinality estimation considering differential privacy protection in related technologies, the calculation speed is often slow due to reasons such as more memory occupation and complex calculation processes. Therefore, privacy protection and real-time publishing of cardinality estimation results cannot be achieved simultaneously. Summary of the Invention

[0005] In view of this, one or more embodiments of this specification provide a method and apparatus for determining prefix sum, an electronic device, and a storage medium.

[0006] To achieve the above object, one or more embodiments of this specification provide the following technical solutions:

[0007] According to a first aspect of one or more embodiments of this specification, a method for determining prefix sum is proposed, and the method includes:

[0008] Convert the timestamp of the current data element in the data stream into a binary number;

[0009] Update the base array based on the digital elements corresponding to the data elements in the digital stream and the binary number, and update the noise array based on the binary number, where the digital elements in the digital stream correspond one-to-one with the data elements in the data stream, each element of the base array is used to represent the base of the nodes in a layer of the binary tree, and each element of the noise array is used to represent the noise of the nodes in a layer of the binary tree;

[0010] Determine the prefix sum of the digital stream based on the binary number, the base array, and the noise array, where the prefix sum of the digital stream is used to determine the base estimate value of the data stream.

[0011] In a possible embodiment of this specification, the updating the base array based on the digital elements corresponding to the data elements in the digital stream and the binary number includes:

[0012] Update the i-th element in the base array to the sum of the first i - 1 elements of the base array and the digital elements corresponding to the data elements in the digital stream, where i is the order of the first non-0 element of the binary number;

[0013] Set the first i - 1 elements in the base array to 0.

[0014] In a possible embodiment of this specification, the updating the noise array based on the binary number includes:

[0015] Perform the following operations on each of the first i - 1 elements in the noise array in sequence: generate a random number, and use the sum of the element and the temporary node as the noisy element, and update the temporary node to the weighted sum result of the random number and the noisy element, where the initial value of the temporary node is a random number;

[0016] Set the first i - 1 elements in the noise array to 0, and update the i-th element in the noise array to the temporary node.

[0017] In a possible embodiment of this specification, each generated random number conforms to the Laplace distribution, and each generated random number is independent of each other.

[0018] In a possible embodiment of this specification, the weighted sum result of the random number and the noisy element is obtained in the following manner:

[0019] Determine the weight of the noisy element and the weight of the random number based on the order of the element in the noise array.

[0020] In a possible embodiment of this specification, the determining the prefix sum of the digital stream based on the binary number, the base array, and the noise array includes:

[0021] Take the sum of the elements in the base array that have the same order as each non-zero element of the binary number, and the sum of the elements in the noise array that have the same order as each non-zero element of the binary number, as the prefix sum of the digital stream.

[0022] In a possible embodiment of this specification, the method further includes:

[0023] Determine the bit element corresponding to the data element in the two-dimensional bit array according to the hash value of the data element;

[0024] If the value of the bit element corresponding to the data element is 0, set the bit element to 1, and add the digital element corresponding to the data element to the digital stream, and set the digital element to 1;

[0025] If the value of the bit element corresponding to the data element is 1, keep the bit element unchanged, and add the digital element corresponding to the data element to the digital stream, and set the digital element to 0.

[0026] In a possible embodiment of this specification, the method further includes:

[0027] Determine the cardinality estimate of the data stream according to the number of rows and columns of the two-dimensional bit array, and the prefix sum of the digital stream.

[0028] In a possible embodiment of this specification, the determining the cardinality estimate of the data stream according to the number of rows and columns of the two-dimensional bit array, and the prefix sum of the digital stream, includes:

[0029] If the prefix sum of the digital stream is not greater than the approximation threshold, determine the cardinality estimate of the data stream according to the number of rows of the two-dimensional bit array and the prefix sum of the digital stream;

[0030] If the prefix sum of the digital stream is greater than the approximation threshold, determine the cardinality estimate of the data stream according to the number of rows and columns of the two-dimensional bit array, the prefix sum of the digital stream, and the Euler-Mascheroni constant.

[0031] In a possible embodiment of this specification, the approximation threshold is the prefix sum of the digital stream when the cardinality estimate of the data stream is ;

[0032] The determining the cardinality estimate of the data stream according to the number of rows of the two-dimensional bit array and the prefix sum of the digital stream, includes:

[0033] Determine the cardinality estimate of the data stream according to the following formula:

[0034]

[0035] In the above formula, m is the number of rows of the two-dimensional bit array, is the prefix sum of the digital stream.

[0036] In a possible embodiment of the present specification, the approximate threshold is the prefix sum of the digital stream when the cardinality estimate value of the data stream is ;

[0037] Determining the cardinality estimate value of the data stream according to the number of rows and columns of the two-dimensional bit array, the prefix sum of the digital stream, and the Euler-Mascheroni constant includes:

[0038]

[0039] In the above formula, m is the number of rows of the two-dimensional bit array, w is the number of columns of the two-dimensional bit array, is the prefix sum of the digital stream, and γ is the Euler-Mascheroni constant.

[0040] According to a second aspect of one or more embodiments of the present specification, a prefix sum determination device is provided, and the device includes:

[0041] A conversion module, configured to convert the timestamp of the current data element in the data stream into a binary number;

[0042] An update module, configured to update the cardinality array based on the digital element corresponding to the data element in the digital stream and the binary number, and update the noise array based on the binary number, wherein the digital elements in the digital stream correspond to the data elements in the data stream one by one, each element of the cardinality array is used to represent the cardinality of a layer of nodes of a binary tree, and each element of the noise array is used to represent the noise of a layer of nodes of a binary tree;

[0043] A determination module, configured to determine the prefix sum of the digital stream based on the binary number, the cardinality array, and the noise array, wherein the prefix sum of the digital stream is used to determine the cardinality estimate value of the data stream.

[0044] According to a third aspect of one or more embodiments of the present specification, a computer program product is provided, including computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0045] According to a fourth aspect of one or more embodiments of the present specification, an electronic device is provided, including:

[0046] A processor;

[0047] A memory for storing processor-executable instructions;

[0048] Wherein, the processor runs the executable instructions to implement the method described in the first aspect.

[0049] According to a fifth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0050] The technical solutions provided by the embodiments of the present specification may include the following beneficial effects:

[0051] For the prefix sum determination method provided by the embodiments of the present specification, the timestamp of the current data element in the data stream is converted into a binary number; based on the digital element corresponding to the data element in the digital stream and the binary number, the base array is updated, and the noise array is updated based on the binary number; based on the binary number, the base array, and the noise array, the prefix sum of the digital stream is determined, wherein the prefix sum of the digital stream is used to determine the base estimate value of the data stream. Since the binary number converted from the timestamp of the data element can represent the node state on the binary tree corresponding to the digital stream, that is, the layer where the nodes included in the prefix sum of the digital stream are located, etc., and each element of the base array is used to represent the base of a layer of nodes of the binary tree, and each element of the noise array is used to represent the noise of a layer of nodes of the binary tree, the prefix sum finally determined based on the above binary number, base array, and noise array is equivalent to the prefix sum determined based on the binary tree corresponding to the digital stream, with a simple calculation method, less memory occupancy, and faster calculation speed; and differential privacy protection can be performed by adding noise, so that the privacy protection and real-time release of the base estimate result are realized simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flowchart of a prefix sum determination method provided by an exemplary embodiment.

[0053] Figure 2 is a schematic diagram of a method for determining a digital element provided by an exemplary embodiment.

[0054] Figure 3 is a schematic diagram of a binary tree provided by an exemplary embodiment.

[0055] Figure 4 is a schematic diagram of a base estimation method provided by an exemplary embodiment.

[0056] Figure 5 is a schematic diagram of the structure of a device provided by an exemplary embodiment.

[0057] Figure 6 It is a block diagram of a prefix sum determination device provided by an exemplary embodiment. Detailed implementation manners

[0058] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with one or more embodiments of this specification. On the contrary, they are merely examples of devices and methods that are consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0059] It should be noted that: in other embodiments, the steps of the corresponding methods are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0060] First, some concepts involved in this specification will be explained.

[0061] Differential privacy: Differential privacy is a technology used to protect user privacy during the process of publishing query information of a database. Its design concept is based on the idea that the best way to protect the privacy of a user is to delete the data entry corresponding to the user so that the user's data does not participate in the calculation of the query. However, if all user data is deleted, the query result will become meaningless. To ensure that the query result is meaningful, we define the concept of differential privacy. By cleverly designing the algorithm, for any two adjacent databases (adjacent means that these two databases differ only in one piece of data), the probability distributions of their query results are similar. If an algorithm is differentially private, then the observer of the query result will not be able to infer whether any user's data is in the database, thus protecting the user privacy in the database.

[0062] Specifically, the definition of differential privacy is as follows: Let ε be a positive real number, and algorithm A be a random algorithm that takes database D as input and outputs A(D). If for any two adjacent databases D1 and D2 that differ only in one record, and for any subset S in the range range(A) of algorithm A, the following inequality always holds:

[0063] P[A(D1) ∈ S] ≤ exp(ε) · P[A(D2) ∈ S]

[0064] Then, the algorithm A is considered to meet the requirements of μ-differential privacy.

[0065] Differential Privacy of Data Streams: In real-world scenarios, it is often necessary to monitor the statistical information of dynamically changing data streams in real time. Many big data scenarios in practice can be abstracted as data streams. For example, the number of viewers who have watched a video is constantly increasing. We can abstract the viewer IDs who have watched a video into a data stream composed of viewer IDs. In the data stream scenario, each data appears only once. If we store all the data that has appeared, it will consume a large amount of storage resources. Therefore, for the real-time monitoring of the statistical information of data streams, people are more concerned about algorithms that consume memory resources sub-linearly (compared to the length of the data stream). To protect user privacy in the data stream scenario, we define the differential privacy of data streams, also known as continuous release differential privacy: Let μ be a positive real number, and the algorithm is a random algorithm that takes a data stream (the length of the data stream is T, and each element is an element in the set U) as input. Denote the output result of the algorithm as (in the data stream, for each incoming element, the algorithm outputs a monitoring result of statistical information based on the current element and the previously observed elements). If for any two adjacent data streams that differ only at one moment for the algorithm the value range of any subset the following inequality holds, then the algorithm is said to provide differential privacy with parameter μ, that is, μ-differential privacy:

[0066]

[0067] Cardinality Estimation: Cardinality is also called Number of Distinct Value (NDV). Cardinality estimation refers to estimating the number of different elements in a data set. This statistic plays a fundamental and important role in many scenarios such as network traffic statistics and database query optimization. In a database, the SQL statement for querying cardinality is as follows:

[0068] SELECT COUNT(DISTINCT col_name)FROM table_name.

[0069] Based on the above technical problems mentioned in the background art, at least one embodiment of this specification provides a prefix sum determination method. This method can perform privacy protection through differential privacy when performing real-time cardinality statistics on a data stream, and simplify the calculation process in the prefix sum determination link of cardinality statistics, reduce memory occupancy, and improve calculation speed, so as to ensure both the privacy protection of the data stream and the real-time release of the cardinality estimation result.

[0070] This method can be executed by a computing device. The computing device can be a terminal device, such as a desktop computer, a portable computer, a tablet computer, a laptop computer, a smart phone, etc. Or, the computing device can be a server, such as a single server, multiple servers, a server cluster, a cloud computing platform, etc. This specification does not limit the type of the computing device.

[0071] It should be noted that in the following content, the order of elements in arrays such as data stream, digital stream, cardinality array, noise array, etc. is from right to left, that is, from the last digit forward.

[0072] Please refer to the appendix Figure 1 , which exemplarily shows the flowchart of this prefix sum determination method, including step S101 to step S103.

[0073] In step S101, convert the timestamp of the current data element in the data stream into a binary number.

[0074] Among them, at any moment t ∈ {1, 2, 3, …, T}, a new data element will be added to the data stream. When any moment t ∈ {1, 2, 3, …, T} is the current moment, t is the timestamp of the current data element.

[0075] In step S102, update the cardinality array based on the digital element corresponding to the data element in the digital stream and the binary number, and update the noise array based on the binary number, where the digital elements in the digital stream correspond one-to-one with the data elements in the data stream, each element of the cardinality array is used to represent the cardinality of a layer of nodes of a binary tree, and each element of the noise array is used to represent the noise of a layer of nodes of a binary tree.

[0076] The following introduces the digital stream.

[0077] Each element in the digital stream is 0 or 1. Compared with the data stream itself, its corresponding digital stream is convenient for subsequent privacy protection mechanisms and real-time cardinality estimation. Every time a data element is added to the data stream, a corresponding digital element can be added to the digital stream for the newly added data element in the manner shown in the appendix Figure 2 as follows:

[0078] First, determine the bit element corresponding to the data element within the two-dimensional bit array according to the hash value of the data element.

[0079] Among them, the two-dimensional bit array can be denoted as B. Denote its number of rows as m and the number of columns as w, and each bit of it is initialized to 0.

[0080] Among them, the current data element is denoted as e t 。

[0081] For example, hash e t and select the element at a position from B as the bit element corresponding to e t Denote the row number of this position as i and the column number as j. It is required that this hash satisfies the following two conditions: the probability that the row number i is equal to {1, 2, …, m} is equal; the probability that the column number j is equal to {1, 2, …, w - 1} is respectively and the probability that the column number j is equal to w is

[0082] Preferably, m = 2 r is an integer power of 2. In this case, the above probability requirements can be achieved in the following way: hash e t into a string of 0s and 1s with a length of r + w - 1, and the probability of each position being 0 or 1 is equal. According to the first r bits (encoded as binary digits), a number that is uniformly distributed between 0 and m - 1 can be obtained, and adding 1 can get the row number i; for the last w - 1 bits, calculate the number of trailing 0s, and adding 1 to this number can get the column number j. For example, if the last w - 1 bits are "1…100", the column number j is taken as 3. The row and column numbers obtained by the above method satisfy the above-mentioned probability distribution requirements.

[0083] Next, if the value of the bit element corresponding to the data element is 0, set the bit element to 1, and add the digital element corresponding to the data element to the digital stream and set the digital element to 1; if the value of the bit element corresponding to the data element is 1, keep the bit element unchanged, and add the digital element corresponding to the data element to the digital stream and set the digital element to 0.

[0084] For example, denote the digital element of the digital stream as b t 。

[0085] Therefore, the data structures of the two-dimensional bit array and the digital stream can be used to statistically calculate the cardinality of ultra-large-scale data with very small memory overhead and time overhead.

[0086] In the related art, a binary tree mechanism is used to determine the prefix sum of the digital stream under the condition of protecting differential privacy, that is, the number Z of 1s appearing in the digital streamt Approximate value Next, a detailed introduction is given to the binary tree mechanism for determining the prefix sum of a digital stream in the related art.

[0087] As Figure 3 shown, first, a binary tree is established, and each leaf node of the binary tree corresponds to an element in the digital stream, and its value is the value of the element; then, the value of each remaining node is equal to the sum of the values of its 2 lower nodes (i.e., its child nodes). Finally, an independent noise is added to each node, and each independent noise follows a Laplace distribution: In the binary tree mechanism, at each moment t, at most nodes are selected, and then they are summed to obtain the prefix sum of the digital stream at that moment. For example, in Figure 3 , to calculate the prefix sum at time t = 7, the first node in the third layer, the third node in the second layer, and the seventh node in the first layer can be selected, and the sum is the prefix sum at time t = 7. However, the binary tree mechanism can be further improved to enhance its accuracy. The improved one is called the optimized binary tree mechanism. The optimized binary tree mechanism reduces the error by weighted averaging multiple random numbers. For example, to calculate the prefix sum at time t = 2, the binary tree only needs to select the first node in the second layer, but in the optimized binary tree mechanism, not only the first node in the second layer needs to be selected, but also the sum of the first two nodes in the first layer needs to be added, and the sum of these two nodes is also an estimated value of the prefix sum at time t = 2. Then, by weighted averaging these two estimated values, a lower error prefix sum estimation can be achieved. However, the optimized binary tree mechanism has a very high computational complexity, and many additions and weighted averaging operations need to be performed at each moment.

[0088] Next, an implementation manner of this step is introduced through an embodiment.

[0089] In this embodiment, when updating the base array based on the digital element corresponding to the data element in the digital stream and the binary number, it can be executed as follows:

[0090] First, update the i-th element in the base array to the sum of the first i - 1 elements in the base array and the digital element corresponding to the data element in the digital stream, where i is the order of the first non-0 element of the binary number.

[0091] For example, the binary number of the timestamp t of the current data element is (μ H …μ2μ1)2, and i = min{k: μ k = 1}. In this step, the i-th element α[i] in the base array α is updated to α[k] is the k-th element in the base array α, bt a digital element corresponding to the current data element within the digital stream.

[0092] Next, set the first i - 1 elements in the base array to 0.

[0093] It should be understood that when i is greater than 1, it is executed in the above - mentioned manner; when i = 1, only the i - th element α[1] in the base array α is updated to b. t That's all.

[0094] In this embodiment, when updating the noise array based on the binary number, it can be executed as follows:

[0095] First, perform the following operations on each of the first i - 1 elements in the noise array in sequence: generate a random number, and use the sum of the element and the temporary node as the noisy element, and update the temporary node to the weighted sum result of the random number and the noisy element, where the initial value of the temporary node is a random number.

[0096] For example, each generated random number conforms to Laplace, and each generated random number is independent of each other.

[0097] For example, the weighted sum result of the random number and the noisy element is obtained in the following manner: determine the weight of the noisy element and the weight of the random number based on the order of the element in the noise array. For example, the weight of the noisy element is: The weight of the random number is where k is the order of the element in the noise array.

[0098] Specifically, it can be decomposed into:

[0099] 1. Initialize the temporary node: where

[0100] 2. For all integers k satisfying 0 < k < i, perform the following operations in ascending order:

[0101] 2.1 Update the k - th element β[k] of the noise array β to β[k]+temp node;

[0102] 2.2 Update temp node to where

[0103] It should be understood that if there is no integer k satisfying 0 < k < i, step 2 is not executed.

[0104] Next, set the first i - 1 elements of the noise array to 0, and update the i - th element of the noise array to the temporary node.

[0105] In step S103, determine the prefix sum of the digital stream based on the binary number, the radix array, and the noise array, where the prefix sum of the digital stream is used to determine the radix estimate value of the data stream.

[0106] For example, take the sum of the elements in the radix array that have the same order as each non-zero element of the binary number, and the sum of the elements in the noise array that have the same order as each non-zero element of the binary number as the prefix sum of the digital stream. That is, determine the prefix sum of the digital stream according to the following formula

[0107]

[0108] In summary, it can be seen that the binary number of the timestamp of the current data element can equivalently represent the layer where the nodes used for summation are located when calculating the prefix sum by the binary tree mechanism. If the k-th bit of this binary number is 1, then the k-th layer of the binary tree has nodes used for summation when calculating the prefix sum; and each element of the radix array is used to represent the radix of the nodes in a layer of the binary tree. That is, if the k-th element of the radix array is non-zero, then the sum of the radixes in the nodes used for summation in the k-th layer of the binary tree is the value of this element. If the k-th element of the radix array is 0, then there are no nodes used for summation in the k-th layer of the binary tree; each element of the noise array is used to represent the noise of the nodes in a layer of the binary tree. That is, if the k-th element of the noise array is non-zero, then the sum of the noises in the nodes used for summation in the k-th layer of the binary tree is the value of this element. If the k-th element of the noise array is 0, then there are no nodes used for summation in the k-th layer of the binary tree. It should be particularly noted that the method of updating the noise array in the above embodiment can be equivalently regarded as the weighted average of the noise in the optimized binary tree mechanism. Therefore, this method equivalently completes the optimized binary tree mechanism, and has simple calculation and fast calculation speed.

[0109] The prefix sum determination method provided by the embodiments of this specification converts the timestamp of the current data element in the data stream into a binary number; updates the base array based on the digital element corresponding to the data element in the digital stream and the binary number, and updates the noise array based on the binary number; determines the prefix sum of the digital stream based on the binary number, the base array, and the noise array, where the prefix sum of the digital stream is used to determine the cardinality estimate of the data stream. Since the binary number converted from the timestamp of the data element can represent the node state on the binary tree corresponding to the digital stream, that is, the layer where the node included in the prefix sum of the digital stream is located, etc., and each element of the base array is used to represent the cardinality of a layer of nodes of the binary tree, and each element of the noise array is used to represent the noise of a layer of nodes of the binary tree, the prefix sum finally determined based on the above binary number, base array, and noise array is equivalent to the prefix sum determined based on the binary tree corresponding to the digital stream. Its calculation method is simple, with less memory occupancy and faster calculation speed; and differential privacy protection can be achieved by adding noise, so that the privacy protection and real-time release of the cardinality estimation result can be realized simultaneously.

[0110] In some embodiments of the present disclosure, the method further includes: determining the cardinality estimate of the data stream according to the number of rows and columns of the two-dimensional bit array, and the prefix sum of the digital stream.

[0111] Exemplarily, if the prefix sum of the digital stream is not greater than the approximation threshold, the cardinality estimate of the data stream is determined according to the number of rows of the two-dimensional bit array and the prefix sum of the digital stream.

[0112] For example, the approximation threshold is the prefix sum of the digital stream when the cardinality estimate of the data stream is ;

[0113] Then the cardinality estimate of the data stream can be determined according to the following formula:

[0114]

[0115] In the above formula, m is the number of rows of the two-dimensional bit array, is the prefix sum of the digital stream.

[0116] Exemplarily, if the prefix sum of the digital stream is greater than the approximation threshold, the cardinality estimate of the data stream is determined according to the number of rows and columns of the two-dimensional bit array, the prefix sum of the digital stream, and the Euler-Mascheroni constant.

[0117] For example, the approximation threshold is the prefix sum of the digital stream when the cardinality estimate of the data stream is ;

[0118] Then, the cardinality estimate value of the data stream can be determined according to the following formula:

[0119]

[0120] In the above formula, m is the number of rows of the two-dimensional bit array, w is the number of columns of the two-dimensional bit array, is the prefix sum of the digital stream, and γ is the Euler-Mascheroni constant.

[0121] Next, the rationality of the above method for calculating the cardinality estimate value of the data stream will be explained.

[0122] The cardinality can be estimated by using the number of bits with value 1 in the two-dimensional bit array B. The reasons are as follows: First, duplicate elements will not cause a change in the number of bits with value 1 in B, because the positions hashed by duplicate elements must have been set to 1, and duplicate elements will not modify B or cause a change in the number of bits with value 1 in B. Second, a larger cardinality often leads to more bits in B being set to 1. Therefore, the number of bits with value 1 in B, denoted as Z t can be used to estimate the cardinality. Assume that the cardinality of the data elements in the data set at the current time t is n t , and the expected value of the number of bits with value 1 in B, denoted as Z t , can be calculated as follows:

[0123]

[0124] After converting the data stream into a digital stream, the prefix sum of the digital stream is exactly Z t . To protect differential privacy, noise is added to the prefix sum of the digital stream to obtain Essentially, at each moment, multiple independent random numbers following t distribution are added to Z , and the expectations of these random numbers are all 0. Therefore, the expected value of the prefix sum of the digital stream after adding noise (or the number of bits with value 1 in the two-dimensional bit array B) is equal to the expected value of Z t :

[0125]

[0126] Therefore, introducing the idea of the first-order moment method estimation in mathematical statistics, the estimated value of the cardinality n of the elements in the data set at the current time t can be deduced from the observed estimated value of the prefix sum t : Denote the inverse function of the function f(·) as g(·). The estimated value of n t can be calculated according to the following formula: ​

[0127]

[0128] However, there is no explicit expression for the function g(·), so it cannot be directly calculated. Previous methods used the binary search method or the Newton iteration method for calculation, but they were time-consuming and caused a serious time-delay bottleneck. This specification proposes an innovative high-accuracy approximation scheme to accurately fit the function g(·), so that the cardinality can be quickly calculated without loss of accuracy.

[0129] First, denote The value of the summation formula can be approximated by an integral:

[0130]

[0131] where the function approaches 0 when u > 2. In addition, the function E1(u) can also be approximated by a Taylor expansion: where the Euler-Mascheroni constant γ ≈ 0.5772.

[0132] When at this time, can be approximated as 0, can be substituted into the Taylor expansion for approximation. Since the independent variable of the function is very small, discarding all polynomial terms during approximation can also achieve extremely high accuracy. At this time Therefore,

[0133]

[0134] Therefore, it can be calculated that:

[0135] When at this time, use the Taylor expansion to approximate Retaining the polynomial terms up to the quadratic term, we can get:

[0136]

[0137] It can be obtained that Therefore, it can be calculated that:

[0138]

[0139] In summary, the function can be approximated by the following explicit expression:

[0140]

[0141] It should be understood that the method for determining the cardinality estimate value of the data stream based on the prefix sum of the digital stream introduced in this embodiment can not only determine the cardinality estimate value based on the prefix sum determined by the prefix sum determination method proposed in this specification, but also determine the cardinality estimate value based on the prefix sum of the digital stream determined by other methods in this field.

[0142] Please refer to the attached Figure 4 , which exemplarily shows a schematic diagram of the cardinality estimation method obtained based on the above various embodiments of this specification.

[0143] First, represent the current timestamp t in binary as t = (μ H …μ2μ1), and find the subscript i of the lowest 1, i = min{k: μ k = 1}.

[0144] Next, update the cardinality array α: α[i] = b t +∑ k＜i α[k];

[0145] Then, create a temporary node where

[0146] Then, for all integers k satisfying 0 < k < i, perform the following operations in ascending order:

[0147] Update the k-th element β[k] of the noise array β to β[k] + temp_node;

[0148] Update temp_node to where

[0149] Set both α[k] and β[k] to 0.

[0150] Then, update the i-th element β[i] of the noise array to the temporary node temp_node, and calculate the prefix sum of the digital stream:

[0151] Finally, quickly map the prefix sum of the digital stream to the cardinality estimate value

[0152] This specification proposes effective optimization solutions for some bottlenecks in the problem of continuous monitoring of cardinality estimation for protecting differential privacy, including an efficient implementation of an optimized binary tree mechanism and a high-precision approximation of the cardinality estimation function. The efficient implementation of the optimized binary tree mechanism is achieved by cleverly constructing and updating temporary nodes, and the high-precision approximation of the cardinality estimation function is achieved by cleverly using integration to approximate the accumulation of complex expressions. With these optimizations, when continuously monitoring the cardinality estimation of differential privacy for a data stream, for each element in the data stream, it can be implemented with an average time complexity of P(1), and the memory space occupied is only about a few KB to dozens of KB.

[0153] Figure 5 is a schematic structural diagram of a device provided by an exemplary embodiment. Please refer to Figure 5 , at the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510. Of course, it may also include other hardware required for other tasks. One or more embodiments of this specification can be implemented based on software. For example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into the memory 508 and then runs it. Of course, in addition to the software implementation method, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0154] Please refer to Figure 6 , the prefix sum determination device can be applied to a device as shown in Figure 5 to implement the technical solution of this specification. The prefix sum determination device can include:

[0155] A conversion module 601 for converting the timestamp of the current data element in the data stream into a binary number;

[0156] An update module 602 for updating the cardinality array based on the digital element corresponding to the data element in the digital stream and the binary number, and updating the noise array based on the binary number, where the digital elements in the digital stream correspond one-to-one with the data elements in the data stream, each element of the cardinality array is used to represent the cardinality of a layer of nodes of the binary tree, and each element of the noise array is used to represent the noise of a layer of nodes of the binary tree;

[0157] A determination module for determining the prefix sum of the digital stream based on the binary number, the cardinality array, and the noise array, where the prefix sum of the digital stream is used to determine the cardinality estimation value of the data stream.

[0158] In a possible embodiment of the present specification, when the update module is used to update the base number array based on the digital elements corresponding to the data elements in the digital stream and the binary number, it is used for:

[0159] Update the i-th element in the base number array to the sum of the first i - 1 elements of the base number array and the digital elements corresponding to the data elements in the digital stream, where i is the order of the first non-0 element of the binary number;

[0160] Set the first i - 1 elements in the base number array to 0.

[0161] In a possible embodiment of the present specification, when the update module is used to update the noise array based on the binary number, it is used for:

[0162] Perform the following operations on each of the first i - 1 elements of the noise array in sequence: generate a random number, and use the sum of the element and the temporary node as the noisy element, and update the temporary node to the weighted sum result of the random number and the noisy element, where the initial value of the temporary node is a random number;

[0163] Set the first i - 1 elements of the noise array to 0, and update the i-th element of the noise array to the temporary node.

[0164] In a possible embodiment of the present specification, each generated random number conforms to the Laplace distribution, and each generated random number is independent of each other.

[0165] In a possible embodiment of the present specification, the weighted sum result of the random number and the noisy element is obtained in the following manner:

[0166] Determine the weight of the noisy element and the weight of the random number based on the order of the element in the noise array.

[0167] In a possible embodiment of the present specification, the determination module is used for:

[0168] Take the sum result of the elements in the base number array with the same order as each non-0 element of the binary number and the elements in the noise array with the same order as each non-0 element of the binary number as the prefix sum of the digital stream.

[0169] In a possible embodiment of the present specification, the device further includes a digital element module, which is used for:

[0170] Determine the bit element corresponding to the data element in the two-dimensional bit array according to the hash value of the data element;

[0171] If the value of the bit element corresponding to the data element is 0, set the bit element to 1, add a digital element corresponding to the data element to the digital stream, and set the digital element to 1;

[0172] If the value of the bit element corresponding to the data element is 1, keep the bit element unchanged, add a digital element corresponding to the data element to the digital stream, and set the digital element to 0.

[0173] In a possible embodiment of the present specification, the device further includes a cardinality estimation module for:

[0174] Determine the cardinality estimation value of the data stream according to the number of rows and columns of the two-dimensional bit array, and the prefix sum of the digital stream.

[0175] In a possible embodiment of the present specification, the cardinality estimation module is used for:

[0176] If the prefix sum of the digital stream is not greater than the approximation threshold, determine the cardinality estimation value of the data stream according to the number of rows of the two-dimensional bit array and the prefix sum of the digital stream;

[0177] If the prefix sum of the digital stream is greater than the approximation threshold, determine the cardinality estimation value of the data stream according to the number of rows and columns of the two-dimensional bit array, the prefix sum of the digital stream, and the Euler-Mascheroni constant.

[0178] In a possible embodiment of the present specification, the approximation threshold is the prefix sum of the digital stream when the cardinality estimation value of the data stream is ;

[0179] When the cardinality estimation module is used to determine the cardinality estimation value of the data stream according to the number of rows of the two-dimensional bit array and the prefix sum of the digital stream, it is used for:

[0180] Determine the cardinality estimation value of the data stream according to the following formula:

[0181]

[0182] In the above formula, m is the number of rows of the two-dimensional bit array, is the prefix sum of the digital stream.

[0183] In a possible embodiment of the present specification, the approximation threshold is the prefix sum of the digital stream when the cardinality estimation value of the data stream is ;

[0184] When the cardinality estimation module is used to determine the cardinality estimation value of the data stream based on the number of rows and columns of the two-dimensional bit array, the prefix sum of the digital stream, and the Euler-Mascheroni constant, it is used for:

[0185]

[0186] In the above formula, m is the number of rows of the two-dimensional bit array, w is the number of columns of the two-dimensional bit array, is the prefix sum of the digital stream, and γ is the Euler-Mascheroni constant.

[0187] One or more embodiments of this specification also propose a computer program product, including a computer program / instructions. When the computing program / instructions are executed by a processor, the steps of the method provided in the first aspect are implemented.

[0188] One or more embodiments of this specification also propose a computer-readable storage medium, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0189] The systems, devices, modules, or units illustrated in the above embodiments may be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any several of these devices.

[0190] In a typical configuration, a computer includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0191] The memory may include non-permanent memory in the computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0192] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0193] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0194] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0195] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0196] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this specification are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0197] It should be understood that although the terms first, second, third, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0198] The above description is only the preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of one or more embodiments of this specification shall be included within the scope protected by one or more embodiments of this specification.

Claims

1. A prefix sum determination method, the method comprising: Converting the timestamp of the current data element in the data stream into a binary number; Updating a radix array based on the digital element corresponding to the data element in the digital stream and the binary number, and updating a noise array based on the binary number, wherein the digital elements in the digital stream correspond one-to-one with the data elements in the data stream, each element of the radix array is used to represent the radix of a layer of nodes of a binary tree, and each element of the noise array is used to represent the noise of a layer of nodes of the binary tree; Determining the prefix sum of the digital stream based on the binary number, the radix array, and the noise array, wherein the prefix sum of the digital stream is used to determine the radix estimation value of the data stream.

2. The prefix sum determination method according to claim 1, wherein the updating the radix array based on the digital element corresponding to the data element in the digital stream and the binary number comprises: Updating the i-th element in the radix array to the sum of the first i - 1 elements of the radix array and the digital element corresponding to the data element in the digital stream, where i is the order of the first non-zero element of the binary number; Setting the first i - 1 elements in the radix array to 0.

3. The prefix sum determination method according to claim 1, wherein the updating the noise array based on the binary number comprises: Performing the following operations on each of the first i - 1 elements of the noise array in sequence: generating a random number, and taking the sum of the element and a temporary node as a noisy element, and updating the temporary node to the weighted sum result of the random number and the noisy element, wherein the initial value of the temporary node is a random number; Setting the first i - 1 elements of the noise array to 0, and updating the i-th element of the noise array to the temporary node.

4. According to the prefix sum determination method of claim 3, each generated random number conforms to a Laplace distribution, and each generated random number is independent of each other.

5. According to the prefix sum determination method of claim 3, the weighted sum result of the random number and the noisy element is obtained in the following manner: Determining the weight of the noisy element and the weight of the random number based on the order of the element in the noise array.

6. The prefix sum determination method according to claim 1, wherein the determining the prefix sum of the digital stream based on the binary number, the radix array, and the noise array comprises: Taking the sum result of the elements in the radix array with the same order as each non-zero element of the binary number and the elements in the noise array with the same order as each non-zero element of the binary number as the prefix sum of the digital stream.

7. The prefix sum determination method according to claim 1, the method further comprising: Determining the bit element corresponding to the data element in a two-dimensional bit array according to the hash value of the data element; If the value of the bit element corresponding to the data element is 0, setting the bit element to 1, and adding the digital element corresponding to the data element to the digital stream and setting the digital element to 1; If the value of the bit element corresponding to the data element is 1, keep the bit element unchanged, add a digital element corresponding to the data element to the digital stream, and set the digital element to 0.

8. The prefix sum determination method according to claim 7, the method further comprising: Determine an estimated value of the cardinality of the data stream according to the number of rows and columns of the two-dimensional bit array and the prefix sum of the digital stream.

9. The prefix sum determination method according to claim 8, the determining an estimated value of the cardinality of the data stream according to the number of rows and columns of the two-dimensional bit array and the prefix sum of the digital stream, comprising: If the prefix sum of the digital stream is not greater than the approximation threshold, determine an estimated value of the cardinality of the data stream according to the number of rows of the two-dimensional bit array and the prefix sum of the digital stream; If the prefix sum of the digital stream is greater than the approximation threshold, determine an estimated value of the cardinality of the data stream according to the number of rows and columns of the two-dimensional bit array, the prefix sum of the digital stream, and the Euler-Mascheroni constant.

10. The prefix sum determination method according to claim 9, wherein the approximate threshold is the prefix sum of the digital stream when the cardinality estimate value of the data stream is ; The determining an estimated value of the cardinality of the data stream according to the number of rows of the two-dimensional bit array and the prefix sum of the digital stream, comprising: Determine an estimated value of the cardinality of the data stream according to the following formula: In the above formula, m is the number of rows of the two-dimensional bit array, is the prefix sum of the digital stream.

11. The prefix sum determination method according to claim 9, wherein the approximate threshold is the prefix sum of the digital stream when the cardinality estimate value of the data stream is ; and the prefix sum of the digital stream The determining an estimated value of the cardinality of the data stream according to the number of rows and columns of the two-dimensional bit array, the prefix sum of the digital stream, and the Euler-Mascheroni constant, comprising: In the above formula, m is the number of rows of the two-dimensional bit array, and w is the number of columns of the two-dimensional bit array. is the prefix sum of the digital stream, and γ is the Euler-Mascheroni constant.

12. A prefix sum determination device, the device comprising: A conversion module, configured to convert the timestamp of the current data element in the data stream into a binary number; An update module, configured to update a cardinality array based on a digital element corresponding to the data element in the digital stream and the binary number, and update a noise array based on the binary number, wherein the digital elements in the digital stream correspond to the data elements in the data stream one by one, each element of the cardinality array is used to represent the cardinality of a layer of nodes of a binary tree, and each element of the noise array is used to represent the noise of a layer of nodes of a binary tree; A determination module, configured to determine the prefix sum of the digital stream based on the binary number, the cardinality array, and the noise array, wherein the prefix sum of the digital stream is used to determine an estimated value of the cardinality of the data stream.

13. The prefix sum determination device according to claim 12, when the update module is configured to update the cardinality array based on a digital element corresponding to the data element in the digital stream and the binary number, it is configured to: Update the i-th element in the radix array to the sum of the first i - 1 elements in the radix array and the digital elements corresponding to the data elements in the digital stream, where, i is the order of the first non-0 element of the binary number; Set the first i-1 elements in the cardinality array to 0.

14. The prefix sum determination device according to claim 12, when the update module is configured to update the noise array based on the binary number, it is configured to: Perform the following operations on each of the first i - 1 elements of the noise array in sequence: generate a random number, and use the sum of the element and the temporary node as the noisy element, and update the temporary node to the weighted sum result of the random number and the noisy element, where, The initial value of the temporary node is a random number; Set the first i-1 elements of the noise array to 0, and update the i-th element of the noise array to the temporary node.

15. The prefix sum determination device according to claim 12, the determination module is configured to: The sum of the elements in the base array that have the same order as each non-zero element of the binary number, and the sum of the elements in the noise array that have the same order as each non-zero element of the binary number, is used as the prefix sum of the digital stream.

16. The prefix sum determination device according to claim 12, the device further comprising a digital element module for: Determine the bit element corresponding to the data element in the two-dimensional bit array according to the hash value of the data element; If the value of the bit element corresponding to the data element is 0, set the bit element to 1, and add the digital element corresponding to the data element to the digital stream, and set the digital element to 1; If the value of the bit element corresponding to the data element is 1, keep the bit element unchanged, and add the digital element corresponding to the data element to the digital stream, and set the digital element to 0.

17. The prefix sum determination device according to claim 16, the device further comprising a cardinality estimation module for: If the prefix sum of the digital stream is not greater than the approximate threshold, determine the cardinality estimation value of the data stream according to the number of rows of the two-dimensional bit array and the prefix sum of the digital stream; If the prefix sum of the digital stream is greater than the approximate threshold, determine the cardinality estimation value of the data stream according to the number of rows and columns of the two-dimensional bit array, the prefix sum of the digital stream, and the Euler-Mascheroni constant.

18. The prefix sum determination device according to claim 17, wherein the approximate threshold is the prefix sum of the digital stream when the cardinality estimate value of the data stream is ; and the prefix sum of the digital stream at this time. When the cardinality estimation module is used to determine the cardinality estimation value of the data stream according to the number of rows of the two-dimensional bit array and the prefix sum of the digital stream, it is used for: Determine the cardinality estimation value of the data stream according to the following formula: In the above formula, m is the number of rows of the two-dimensional bit array, is the prefix sum of the digital stream; When the cardinality estimation module is used to determine the cardinality estimation value of the data stream according to the number of rows and columns of the two-dimensional bit array, the prefix sum of the digital stream, and the Euler-Mascheroni constant, it is used for: In the above formula, m is the number of rows of the two-dimensional bit array, and w is the number of columns of the two-dimensional bit array. is the prefix sum of the digital stream, and γ is the Euler-Mascheroni constant.

19. A computer program product, comprising computer programs / instructions, which when executed by a processor implement the steps of the method according to any one of claims 1 to 11.

20. An electronic device, comprising: A processor; A memory for storing processor-executable instructions; Wherein, the processor realizes the method according to any one of claims 1 to 11 by running the executable instructions.

21. A computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are realized.

Images