A sketch-based low-frequency item frequency estimation method under local differential privacy

Abstract

The application relates to the technical field of data privacy protection, and provides a sketch-based low-frequency item frequency estimation method under local differential privacy, which comprises the following steps: a user constructs a multi-layer sketch structure based on a local data set, a generalized random response mechanism is applied to disturb column coordinates of each layer of the sketch, so that a local differential privacy constraint is met; a server receives disturbed row coordinate and column coordinate pairs uploaded by all users, aggregates the disturbed row coordinate and column coordinate pairs into a global multi-layer sketch, constructs a global multi-layer sketch matrix, calibrates observation counts of each layer of the global multi-layer sketch matrix, estimates real frequencies of each layer of the global multi-layer sketch matrix, and based on the calibrated frequency estimation of each layer, a query item is calculated to obtain a global frequency estimation of the query item. By introducing the local differential privacy mechanism and optimizing the sketch structure, the application can significantly improve the frequency estimation precision of low-frequency data items while strictly protecting the individual privacy of the user.

Description

Technical Field

[0001] This invention belongs to the field of data privacy protection technology, and in particular relates to a method for estimating the frequency of low-frequency terms based on sketches under local differential privacy. Background Technology

[0002] Local differential privacy, unlike traditional privacy mechanisms that rely on centralized trusted parties, perturbs and protects information locally at the source of data collection and analysis, effectively reducing the risk of sensitive information leakage at the source. However, in practical applications, some data with small sample sizes are easily overwhelmed by noise, leading to lower estimation accuracy, which can significantly impact the accuracy of global frequency estimation or statistical analysis. But these small sample sizes do not mean the data is worthless. For example, in cybersecurity scenarios, although some edge nodes report less data, the niche events or anomalies they contain are of great significance for overall risk detection.

[0003] Sketching, as an efficient data compression structure, is widely used in streaming data processing and frequency statistics. Sketches can approximate the frequency distribution of large-scale data with relatively small space and communication overhead. Uploading locally generated sketches from each participant to a server for unified aggregation can alleviate the problem of insufficient data from individual participants. However, the sharing and aggregation process still carries the potential risk of privacy breaches. If the communication link or aggregation node is attacked, sensitive information could potentially be deduced from the sketches, posing a privacy threat.

[0004] Numerous frequency estimation techniques based on local differential privacy and sketching have been proposed to achieve efficient data statistics and analysis while protecting data privacy. However, most methods focus more on improving the accuracy of high-frequency terms in practical applications, neglecting the impact of differences in data scale and distribution among different participants. When some participants have small sample sizes or sparse data features, simply relying on fixed noise injection or a uniform sketching strategy often amplifies the estimation error, leading to distorted statistical results for low-frequency terms and affecting the reliability and fairness of the overall analysis.

[0005] Therefore, the problem with existing technologies is that accuracy and privacy are difficult to balance. Increasing noise to enhance privacy protection often further weakens the estimation ability of low-frequency terms, while improving accuracy may sacrifice privacy and security. Existing methods lack an effective balance mechanism between the two. Furthermore, there is a lack of effective protection and aggregation mechanisms in the sketch sharing and aggregation process, which poses potential risks of privacy leakage and insufficient aggregation estimation accuracy. Summary of the Invention

[0006] In view of the above-mentioned deficiencies of the prior art, this invention proposes a sketch-based low-frequency term frequency estimation method under local differential privacy. The technical solution designed in this invention includes the following steps:

[0007] Each user constructs a multi-layer sketch structure based on a local dataset. Each sketch uses an independent hash function to map data items to row and column coordinates. A generalized random response mechanism is applied to the column coordinates of each sketch to perturb them in order to meet local differential privacy constraints. The perturbed row and column coordinate pairs are then uploaded to the server.

[0008] The server receives all perturbed row and column coordinate pairs uploaded by users and aggregates them into a global multi-layer sketch matrix. It then calibrates the observation counts of each layer of the global multi-layer sketch matrix, estimates the true frequency of each layer, and, based on the calibrated frequency estimates of each layer, calculates the global frequency estimate of the query item through joint computation.

[0009] Preferably, the multi-layer sketch structure includes:

[0010] This is an XY-Sketch sketch used for compressing and indexing data frequency information.

[0011] Preferably, satisfying the local differential privacy constraint includes:

[0012] The privacy budget allocated to each layer is d is the number of sketch layers. For privacy budget parameters.

[0013] Preferably, the perturbation of the column coordinates of each sketch layer using a generalized stochastic response mechanism includes:

[0014] The true column is ct. Users report ct with probability p and any other column with probability q, satisfying local differential privacy constraints. The formulas for probabilities p and q are as follows:

[0015]

[0016]

[0017] In the formula, The number of columns for each layer of the sketch. It is an exponential function.

[0018] Preferably, the global multi-layer sketch matrix includes:

[0019] The global multi-layer sketch matrix is ​​in two-dimensional matrix form, and each cell records the occurrence count of the corresponding row and column coordinates.

[0020] Preferably, the formula for estimating the true frequency of each layer of the global multi-layer sketch matrix is ​​as follows:

[0021]

[0022] In the formula, For the first The true frequency of the layer, For the first Layer Observation count of the column, This represents the total number of participating users.

[0023] Preferably, the global frequency estimate of the query term is obtained through joint calculation, as shown in the following formula:

[0024]

[0025] In the formula, d represents the number of sketch layers.

[0026] Beneficial effects:

[0027] 1. This invention uses a local differential privacy mechanism on the user side to perturb the row and column indexes in the sketch structure, ensuring that the user's original data never leaves the local machine. This method cuts off the path of privacy leakage at the source, making it impossible for the server to infer or deduce the real information of any individual when aggregating data. It achieves quantifiable and provable privacy protection strength, and fundamentally guarantees the security of users when participating in data contributions.

[0028] 2. This invention achieves weighted enhancement and accurate recovery of low-frequency data through an optimized multi-layer sketch structure and adaptive parameter design. By introducing dynamic adjustment and bias correction mechanisms in the aggregation calibration stage, it effectively offsets the huge estimation bias caused by noise injection to low-frequency terms, thereby significantly improving the accuracy and reliability of statistical results for obscure data while protecting privacy.

[0029] 3. This invention establishes a new and more efficient balance between privacy protection strength and data estimation accuracy through ingenious system design. At the cost of limited privacy budget and communication overhead, this method maximizes the extraction of effective statistical information through the compression of sketch structure and the optimization of calibration algorithm, and finally achieves high accuracy and stability of overall frequency estimation, especially low-frequency term estimation, under strict privacy constraints. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the user side and server side of a preferred embodiment of the present invention;

[0031] Figure 2This is a schematic diagram of the user side and server side of a preferred embodiment of the present invention. Detailed Implementation

[0032] The embodiments of the present invention will be described in detail below. The embodiments described below are implemented based on the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the embodiments described below.

[0033] This invention designs a sketch-based low-frequency term frequency estimation method under local differential privacy. The technical solution includes the following steps, such as... Figure 1-2 As shown, it specifically includes:

[0034] Each user constructs a multi-layer sketch structure based on a local dataset. Each sketch uses an independent hash function to map data items to row and column coordinates. A generalized random response mechanism is applied to the column coordinates of each sketch to perturb them in order to meet local differential privacy constraints. The perturbed row and column coordinate pairs are then uploaded to the server.

[0035] The server receives all perturbed row and column coordinate pairs uploaded by users and aggregates them into a global multi-layer sketch matrix. It then calibrates the observation counts of each layer of the global multi-layer sketch matrix, estimates the true frequency of each layer, and, based on the calibrated frequency estimates of each layer, calculates the global frequency estimate of the query item through joint computation.

[0036] Specifically, on the user side, each participant possesses a local dataset and constructs an XY-Sketch structure locally for compressing and indexing data frequency information. Subsequently, a local differential privacy mechanism is used to perturb the row and column information of the sketch to ensure that the user's true data distribution is not leaked during data upload. On the server side, the system receives the perturbed (row, column) data of the sketch from all participants and restores the overall frequency distribution through aggregation and calibration modules, thereby obtaining a global estimation result under privacy protection.

[0037] Preferably, the multi-layer sketch structure includes:

[0038] This is an XY-Sketch sketch used for compressing and indexing data frequency information.

[0039] Preferably, satisfying local differential privacy constraints includes:

[0040] The privacy budget allocated to each layer is d is the number of sketch layers. For privacy budget parameters.

[0041] Preferably, the column coordinates of each sketch layer are perturbed using a generalized stochastic response mechanism, including:

[0042] The true column is ct. Users report ct with probability p and any other column with probability q, satisfying local differential privacy constraints. The formulas for probabilities p and q are as follows:

[0043]

[0044]

[0045] In the formula, The number of columns for each layer of the sketch. It is an exponential function.

[0046] Specifically, for the user-side process, each user selects one or more data items x from the local dataset. i This data is then mapped to two-dimensional coordinates (row, column) in the XY-Sketch. Rows and columns in the sketch are mapped using independent hash functions to ensure randomness and conflict balance in data distribution. To protect privacy, users perturb the index information of each column in the sketch using a generalized random response (GRR) mechanism. Let the true column be ct; then, the user reports ct with probability p and any other column with probability q, satisfying the ε-LDP constraint. The perturbed row-column pair (row*, column*) is the upload object. Users upload the perturbed (row*, column*) pair to the server without containing any original numerical information, thus ensuring individual-level privacy.

[0047] Preferably, the global multi-layer sketch matrix includes:

[0048] The global multi-layer sketch matrix is ​​in two-dimensional matrix form, and each cell records the occurrence count of the corresponding row and column coordinates.

[0049] Preferably, the true frequency of each layer of the global multi-layer sketch matrix is ​​estimated using the following formula:

[0050]

[0051] In the formula, For the first The true frequency of the layer, For the first Layer Observation count of the column, This represents the total number of participating users.

[0052] Preferably, the global frequency estimate of the query term is obtained through joint calculation, as shown in the following formula:

[0053]

[0054] In the formula, d represents the number of sketch layers.

[0055] Specifically, the server receives (row*, column*) pairs from n users and fills them into a global XY-Sketch matrix based on their row and column coordinates. Each cell in the matrix records the frequency of occurrence at the corresponding position. For example, Figure 1 In the example shown on the right, the elements (30, 20, 40, 50) of the matrix represent the perturbation counts in different hash buckets. Since the uploaded data undergoes GRR perturbation, the server needs to calibrate the matrix counts. Finally, the server jointly infers the frequencies based on the joint row and column probabilities.

[0056] In addition, the server identifies abnormal access patterns based on the estimated frequency distribution, marks low-frequency but high-risk events as potential security threats, and generates risk reports. The entire process enables distributed detection of anomalous behavior in large-scale networks without sharing raw traffic logs.

[0057] This embodiment demonstrates the significant advantages of this invention in distributed privacy data analysis. By combining a sketch structure with a local differential privacy mechanism, it can accurately recover the global frequency distribution while protecting individual privacy, exhibiting superior performance, particularly in low-frequency event identification. This method can be applied not only to network security monitoring but also extended to multiple fields such as user behavior statistics, privacy recommendation systems, and IoT anomaly detection, possessing broad practical application value and promotional significance.

[0058] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for estimating the frequency of low-frequency terms based on sketches under local differential privacy, characterized in that, include: Each user constructs a multi-layer sketch structure based on a local dataset. Each sketch uses an independent hash function to map data items to row and column coordinates. A generalized random response mechanism is applied to the column coordinates of each sketch to perturb them in order to meet local differential privacy constraints. The perturbed row and column coordinate pairs are then uploaded to the server. The server receives all perturbed row and column coordinate pairs uploaded by users and aggregates them into a global multi-layer sketch to construct a global multi-layer sketch matrix. It calibrates the observation counts of each layer of the global multi-layer sketch matrix, estimates the true frequency of each layer of the global multi-layer sketch matrix, and obtains the global frequency estimate of the query item through joint calculation based on the frequency estimate after calibration of each layer. The perturbation of the column coordinates of each sketch layer by applying a generalized stochastic response mechanism includes: The true column is ct. Users report ct with probability p and any other column with probability q, satisfying local differential privacy constraints. The formulas for probabilities p and q are as follows: In the formula, The number of columns for each layer of the sketch. It is an exponential function; The formula for estimating the true frequency of each layer of the global multi-layer sketch matrix is ​​as follows: In the formula, For the first The true frequency of the layer, For the first Layer Observation count of the column, Total number of participating users; The global frequency estimate of the query term is obtained through joint calculation, as shown in the following formula: In the formula, d represents the number of sketch layers.

2. The method for estimating the frequency of low-frequency terms based on sketches under local differential privacy as described in claim 1, characterized in that, The multi-layer sketch structure includes: This is an XY-Sketch sketch used for compressing and indexing data frequency information.

3. The method for estimating the frequency of low-frequency terms based on sketches under local differential privacy as described in claim 1, characterized in that, The conditions for satisfying local differential privacy constraints include: The privacy budget allocated to each layer is d is the number of sketch layers. For privacy budget parameters.

4. The method for estimating the frequency of low-frequency terms based on sketches under local differential privacy as described in claim 1, characterized in that, The global multi-layer sketch matrix includes: The global multi-layer sketch matrix is ​​in two-dimensional matrix form, and each cell records the occurrence count of the corresponding row and column coordinates.

Images