A top-k network flow estimation method based on local differential privacy

Abstract

The application discloses a kind of top-k network stream estimation methods based on local differential privacy, by combining with multi-round iteration method, each user is randomly allocated or added to a group, reconstructs the domain of network stream collected in a period of time, corresponding network stream size and network stream cardinality are counted and sent to server end.Server records the user data in different groups, calculates the optimal parameters required for privacy protection and returns to the corresponding client.Each user of the client uses sketch to store the collected network stream, and according to the value of the optimal parameter provided by the server, the data in the sketch is disturbed locally, and the disturbed result is submitted to the server end.Finally, after collecting all the disturbed data of the users, the server aggregates and iteratively estimates the top-k network stream.The application is based on local differential privacy, which further improves the utility of estimating top-k network stream under the premise of protecting privacy.

Description

Technical Field

[0001] This invention belongs to the field of information security technology and relates to a top-k network flow estimation method based on local differential privacy. Background Technology

[0002] With the continuous development of network technology and its applications, the Internet has permeated every aspect of people's lives. In production and daily life, information from all aspects of life needs to be disseminated through network digitization, resulting in a massive amount of network data flow. Network measurement is a fundamental strategy for understanding the dynamics of network infrastructure, providing necessary information for various network management needs, such as load balancing, billing, intrusion detection, and performance diagnostics. Network measurement is crucial for effective network management and daily network operations. Efficient and usable network measurement methods require statistical information on network devices and network flows. Typical statistical information includes network flow size, network flow cardinality, percentage of data volume in a specific network flow, and top-k network flows. Statistical information describes the performance of the network system and provides necessary decision-making basis for network operation and security.

[0003] In real-world scenarios, data flows typically follow skewed distributions such as Zipf or Power-Law. In network measurement, a minority of hot flows have a greater impact on the network, with the k most frequent flows being called the top-k network flows. Estimating the top-k network flows is crucial in network measurement, and many network statistics problems rely on their estimation. Based on the statistics of top-k network flows, network researchers can analyze traffic characteristics and apply them to network tasks such as traffic engineering, anomaly detection, and congestion control. Common methods for statistically analyzing top-k network flows in network measurement can be divided into two types: counting all strategies and counting part strategies. Counting all strategies often use compact, shared data structures, such as sketches, to reduce storage overhead. Counting part strategies, on the other hand, reduce storage overhead by restricting replacement strategies.

[0004] Because network measurement data carries a large amount of sensitive information, it cannot be directly published. Researching reliable and secure privacy protection technologies for network measurement data is an urgent problem to be solved. Privacy protection technologies have gradually evolved from initial methods based on anonymity and encryption to perturbation-based methods. Among these perturbation-based methods, Differential Privacy (DP) has become a standard for privacy data publication, reducing the limitations of the previous two methods. Due to the lack of fully trustworthy third parties in real life, Local Differential Privacy (LDP) has emerged. In LDP, user data is locally perturbed, which not only protects against background knowledge attacks but also prevents data leakage due to untrusted third parties. LDP is now widely used, such as in attacker detection, edge distribution, and deep learning. Summary of the Invention

[0005] Purpose of the invention: This invention provides a top-k network flow estimation method based on local differential privacy, which combines privacy protection technology with network measurement tasks that statistically measure top-k network flows, so as to facilitate wider application in the future.

[0006] To achieve the above-mentioned objectives, the technical solution provided by this invention is as follows.

[0007] A top-k network flow estimation method based on local differential privacy includes the following steps:

[0008] S1. The client performs preprocessing operations;

[0009] Each user R i Collect network flow data over a period of time, using source IP addresses as flow identifiers to distinguish different flows. This is done by randomly assigning or adding a packet G. w (Total Υ groups), calculate group G w Prefix length of network flows Reconstructing Group G w Domain of network flow Calculate the corresponding network flow size n i and network flow cardinality γ i .

[0010] S2. The server records the network flow information of different user groups statistically analyzed in step (S1) and performs preprocessing operations.

[0011] The server first estimates the R value for each user. i The upper bound U of the network flow size iThen, based on the information collected from different groups, the server performs privacy and utility analysis to calculate G for each group, while satisfying the L-neighbor-(ε,δ) local differential privacy model. w The corresponding optimal parameter: optimal number of bins Optimal noise parameters

[0012] S3, The client uses the group G obtained in step (S2) to belong to the group. w Optimal parameters Perform local disturbance operations:

[0013] Each group G w Each user R i First, two hash functions are randomly selected. The network streaming data collected over a period of time is stored using a sketch, and then subjected to the following perturbation:

[0014]

[0015]

[0016] S4. The server aggregates the results of all users after the perturbation in step (S3) and the hash function information used, and estimates the top-k network flow through iteration.

[0017] Furthermore, step (S1) includes the following process:

[0018] The setup consists of one server and multiple client machines. Network traffic can be converted into a set of M types of network flows, F = {f1, f2, ..., f...}. M By setting the source IP address to an IPv4 address, the network flow domain is obtained as D, where d = |D| = 2. 32 f i ∈[0,d-1]. Since multiple network flow packets are likely to belong to the same network flow, each user R... i The size of the network flow is denoted as n i The cardinality of network flows is denoted as γ. i The upper bound of the network flow size is denoted as U. i Accordingly, each user R i The collected network stream data can be converted into a vector, denoted as v. i =(v i,0 ,v i,1 ,…,v i,d-1 )∈{0,1,…U i} d After being added to the corresponding group, each user R i A counter can be used to count ni To reduce storage overhead, users use HyperloglogSketch to estimate γ. i .

[0019] Furthermore, step (S2) includes the following process:

[0020] The server assumes that each user R i The collected network flows follow a Zipf distribution with a skewness of 1, and are set to pass through 0.16n... i After determining the number of network flows, the size of each statistically significant network flow is assigned the value of 1, thus yielding an upper bound for the estimated network flow size.

[0021] Because each user R i Report a vector v i The set of all user input vectors is V∈{0,1,…,U} max} N·d When the inputs V and V' of two user vector sets exist as defined below:

[0022] Define L-adjacent - (ε,δ) local differential privacy as follows: There exists a randomly perturbated mechanism M:X→R such that for any two adjacent inputs V,V'∈X, and |V-V'|1≤L, the output subset of all mechanisms is... The following inequalities must be satisfied:

[0023] Pr[M(V)∈S]≤e ε Pr[M(V′)∈S]+δ

[0024] Where ε is the privacy budget, and δ is the mechanism M that satisfies ε-LDP with a minimum probability of 1-δ.

[0025] The total number of data packets is fixed at n. In a single user's input, at most one data packet's IP address changes to another, resulting in L = 2. Accordingly, the server, through privacy and utility analysis, calculates the value of each packet G under the premise of satisfying the L-adjacent-(ε,δ) local differential privacy model. w Corresponding optimal parameters Finally, the server sends the calculated optimal parameters to each client in the corresponding group.

[0026] Furthermore, step (S3) includes the following process:

[0027] Each group G w Each user R i Network flow data v collected over a period of time i From the original length domain hash to Stored in a long sketch. Each position B in each user's sketch. i,j The value stored in is that network stream data first passes through h i The hash function hashes to position j, then passes through s. i The hash function hashes the network stream to ±1 and then sums the results. Finally, privacy protection is implemented for the data in the sketch by adding... The noise is perturbed to satisfy the defined L-neighbor-(ε,δ) local differential privacy model.

[0028] Furthermore, step (S4) includes the following process:

[0029] The server provides each group G g (g∈{1,2,…,Υ}) Enumerate a fixed length prefix set The server is pre-configured When estimating the frequency of each prefix in the network flow prefix set for each packet, the steps are as follows:

[0030]

[0031] The server will generate a set of top-k network flow prefixes, starting with the first packet G1. Then, the server expands based on the existing set of top-k network flow prefixes, estimating the number of occurrences of all possible expansions of the next set of top-k network flow prefixes. This second step is repeated until the total length limit is reached, and the estimated top-k network flows are finally obtained through iteration.

[0032] Beneficial Effects: This invention addresses the top-k network flow statistics task in network measurement by combining it with local differential privacy for corresponding privacy protection. While satisfying the L-neighborhood-(ε,δ) local differential privacy model, this invention can reasonably protect network flow data while ensuring the security of privacy data, resulting in higher security of privacy data, more stable processing and data operation models, and improved utility for estimating top-k network flows, thus broadening its application. Attached Figure Description

[0033] Figure 1 This is the overall flowchart of the present invention;

[0034] Figure 2 This is a model diagram of a top-k network flow estimation method based on local differential privacy according to the present invention;

[0035] Figure 3 This is a schematic diagram of the disturbance in this invention. Detailed Implementation

[0036] To illustrate the technical solutions disclosed in this invention in detail, further explanation will be provided below with reference to the accompanying drawings and specific embodiments.

[0037] First, addressing the insufficient protection of sensitive data in existing network measurement tasks, this invention defines an L-neighbor-(ε,δ) local differential privacy model. Second, based on this model, a perturbation mechanism for network flow data is designed to protect it. The method described in this invention can be used to estimate top-k network flows and further applied to network tasks such as traffic engineering, anomaly detection, and congestion control.

[0038] In the method used in this invention, the source IP address in the network flow data packet is first set as the flow identifier, and the problem model and local differential privacy model are defined. Combined with a multi-round iterative method, each user is randomly assigned or joined to a group, reconstructing the domain of the network flows collected over a period of time, calculating the corresponding network flow size and cardinality, and sending them to the server. The server records user data in different groups, calculates the optimal parameters required for privacy protection, and returns them to the corresponding clients. Each user on the client uses a sketch to store the collected network flows, perturbs the data in the sketch locally based on the optimal parameter values ​​provided by the server, and submits the perturbated result to the server. Finally, after collecting the perturbed data from all users, the server aggregates the data and iteratively estimates the top-k network flows.

[0039] Specifically, combining Figure 1 A top-k network flow estimation method based on local differential privacy, comprising the following steps:

[0040] (1) For the client, the preprocessing steps are as follows:

[0041] (1.1) Each user R i Collect network data packets over a period of time, using source IP addresses as flow identifiers to distinguish different flows, and denote the total number of users as N.

[0042] Network traffic can be converted into a set of M types of network flows, F = {f1, f2, ..., f...} M By setting the source IP address to an IPv4 address, the network flow domain is obtained as D, where d = |D| = 2. 32 f i ∈[0,d-1]. Since multiple network flow packets are likely to belong to the same network flow, each user R... i The size of the network flow is denoted as n i The cardinality of network flows is denoted as γ. iThe upper bound of the network flow size is denoted as U. i Accordingly, each user R i The collected network stream data can be converted into a vector, denoted as v. i =(v i,0 ,v i,1 ,…,v i,d-1 )∈{0,1,…U i} d .

[0043] (1.2) Each user R i Randomly assigned to or added to a group G w (Total Υ groups), reconstruct the domain of the network flow for the corresponding group, and count the size n of the corresponding network flow. i and network flow cardinality γ i Send it to the server.

[0044] By calculating group G w Prefix length of network flows Reconstructing the domain of network flow After being added to the corresponding group, each user R i A counter can be used to count n i To reduce storage overhead, users use Hyperloglog Sketch to estimate γ. i Finally, the statistical network flow information is sent to the server.

[0045] (2) For the server side, the preprocessing steps required are as follows:

[0046] (2.1) The server collects network flow information from different groups of users and estimates R for each user. i The upper bound U of the network flow size i .

[0047] The server assumes that each user R i The collected network flows follow a Zipf distribution with a skewness of 1, and are set to pass through 0.16n... i After determining the number of network flows, the size of each statistically significant network flow is assigned the value of 1, thus yielding an upper bound for the estimated network flow size.

[0048] (2.2) The server side is based on different groups G w The collected information, through privacy and utility analysis, is used to calculate G for each group, while satisfying the L-neighbor-(ε,δ) local differential privacy model. w The corresponding optimal parameter: optimal number of bins Optimal noise parameters

[0049] Based on the defined L-neighbor-(ε,δ) local differential privacy model, the total number of data packets is fixed at n, and the adjacent input is the input of a single user, where at most one data packet's IP address changes to another IP address, thus obtaining L = 2. Accordingly, the server, through privacy analysis and utility analysis, calculates the value of each packet G under the premise of satisfying the L-neighbor-(ε,δ) local differential privacy model. w Corresponding optimal parameters Finally, the server sends the calculated optimal parameters to each client in the corresponding group.

[0050] (3) For the client, the required local perturbation steps are as follows:

[0051] (3.1) The present invention provides the following perturbation mechanism.

[0052] Combination Figure 2 , Figure 3 Each group G w Each user R i Use optimal parameters Perform a perturbation and send the perturbation result to the server.

[0053] Each group G w Each user R i First, two hash functions are randomly selected. Network flow data v collected over a period of time i From the original length domain hash to Stored in a long sketch. Each position B in each user's sketch. i,j The value stored in is that network stream data first passes through h i The hash function hashes to position j, then passes through s. i The hash function hashes the network stream to ±1 and then sums the results. Finally, privacy protection is implemented for the data in the sketch by adding... The noise is perturbed to satisfy the defined L-neighbor-(ε,δ) local differential privacy model. The process is as follows:

[0054]

[0055]

[0056] (3.2) Finally, each user sends the perturbation result and the hash function information used to the server.

[0057] (4) For the server side, the aggregation and estimation steps required are as follows:

[0058] (4.1) After the server collects the perturbation data of all users in different groups, it aggregates the data.

[0059] The server provides each group G g (g∈{1,2,…,Υ}) Enumerate a fixed length prefix set The server is pre-configured When estimating the frequency of each prefix in the network flow prefix set for each packet, the steps are as follows:

[0060]

[0061] The server will generate a set of top-k network flow prefixes, starting with the first packet G1. Then, the server expands based on the existing set of top-k network flow prefixes, estimating the number of times all possible expansions of the next set of top-k network flow prefixes would occur. This second step is repeated until the total length limit is reached.

[0062] (4.2) Finally, the server can obtain the estimated top-k network flow through iteration.

[0063] Based on the above calculation process, the following are the experimental results of this invention.

[0064] The experiment used both real and synthetic datasets. The real dataset was the CAIDA Anonymous OC48 Internet Tracking 2002-2003 dataset, from which the network flow data for the first second of the data on August 14, 2002, was extracted. It contained 74,904 records, with a network flow cardinality / data sparsity of 8904. Assuming 700 customers, it was divided into 3 groups. The synthetic dataset was generated according to a Zipf distribution with a skewness of 1.01, resulting in a dataset of size 200,000. Data with values ​​less than 10,000 were extracted. The number and cardinality of the filtered data were calculated and mapped one-to-one to the network flow data for synthesis. This dataset contained 93,977 records, with a network flow cardinality / data sparsity of 8383. Assuming 900 customers, it was divided into 3 groups. Two sets of experiments were conducted in total.

[0065] The first set of experiments evaluated the effectiveness of the scheme on real and synthetic datasets by varying the privacy budget ε. The privacy budget ε was set to values ​​of 0.5, 1.0, 1.5, 2.0, 2.5, and 3.0, with δ set to 10. -5 The parameter k was set to 16. The results were evaluated using F1 and NCR scores, and the final F1 and NCR scores of the scheme applied to real and synthetic datasets are shown in Table 1.

[0066] Experimental results show that as the privacy budget ε increases and the level of protection decreases, the F1 and NCR scores of this invention continuously increase, indicating that the accuracy (utility) of the estimated top-k network flows is continuously improving. Under the premise of protecting privacy, the utility of estimating top-k network flows using this invention can be guaranteed to be good.

[0067] Table 1. Utility Evaluation of the Privacy Budget ε Change on Real and Synthetic Datasets

[0068]

[0069]

[0070] The second set of experiments evaluated the effectiveness of the scheme on real and synthetic datasets by varying the parameter k. The parameters k were set to 3, 8, 12, 16, 20, and 25, with δ set to 10. -5 The privacy budget ε was set to 3.0. The results were evaluated using F1 and NCR scores, and the final F1 and NCR scores of the scheme applied to real and synthetic datasets are shown in Table 2.

[0071] The experimental results show that as the parameter k increases, the F1 and NCR scores of this invention follow a trend: first increasing, then stabilizing, and finally decreasing. When the parameter k is small, the top-k network flow can be estimated relatively accurately. However, when the parameter k exceeds a certain value, the utility decreases because it becomes more difficult to estimate infrequent network flows.

[0072] Table 2. Utility evaluation of the scheme on real and synthetic datasets when parameter k is changed.

[0073]

[0074] Finally, it should be noted that existing inventions have not considered privacy protection based on local differential privacy techniques for estimating top-k network flows. This invention proposes a top-k network flow estimation model based on local differential privacy and provides the corresponding problem model and local differential privacy model definition. Furthermore, based on the given definition, a local differential privacy perturbation mechanism for top-k network flow estimation is designed, including providing the corresponding estimation formula. Experiments and data demonstrate that the method described in this invention can achieve finer-grained protection and reasonably protect network flow data while ensuring privacy data security, thereby improving the utility of estimating top-k network flows.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A top- k estimation method based on local differential privacy. A network flow estimation method, characterized in that, Includes the following steps: S1. Client-side preprocessing operations: Each user Collect network flow data over a period of time, and use the source IP address as the flow identifier to identify different flows; Then, by randomly assigning or joining a group... Calculate grouping Prefix length of network flows Reconstructing groups Domain of network flow ; Statistical analysis of the corresponding network flow size and network flow cardinality ; S2. The server records the network flow information of different user groups statistically analyzed in step S1 and performs preprocessing operations: The server first estimates each user Upper bound of network flow size Then, based on the information collected from different groups, the server performs privacy analysis and utility analysis to determine whether the information meets the requirements. Adjacent - Under the premise of local differential privacy model, calculate each group The corresponding optimal parameters include the optimal number of bins. and optimal noise parameters ; The server assumes that each user The collected network flows follow a Zipf distribution with a skewness of 1, and are set to pass through... After determining the number of network flows, the size of each statistically significant network flow is assigned the value of 1, thus yielding an upper bound for the estimated network flow size. ; Due to each user Report a vector The set of all user input vectors is ; When the input of two user vector sets and The following definitions exist: definition Adjacent - Local differential privacy is achieved by the existence of a mechanism for random perturbation. If for any two adjacent inputs ,and A subset of the output of all mechanisms The following inequalities are satisfied: in, For privacy budget, For mechanism With the least The probability satisfies -LDP; The total number of data packets is set to a fixed value. In a given set of adjacent inputs from a single user, at most one data packet's IP address changes to another IP address, thus obtaining... ; Accordingly, the server performs privacy analysis and utility analysis to meet the requirements. Adjacent - Under the premise of local differential privacy model, calculate each group Corresponding optimal parameters , ; Finally, the server sends the calculated optimal parameters to each client in the corresponding group; S3, The client uses the group it belongs to as obtained in step S2. Optimal parameters , Perform local disturbance operations: Each group Each user First, two hash functions are randomly selected. , The network stream data collected over a period of time is stored using a sketch, and then subjected to the following perturbation: ， S4. The server aggregates the results of all users after the perturbation in step S3 and the hash function information used, and iteratively estimates the top-order hash function. Network flow.

2. The top-based local differential privacy top-down method according to claim 1 The network flow estimation method is characterized by, Step S1 includes the following process: The configuration contains one server and more than one client, and network traffic is converted into a single entity. A set of network flows, represented as ; By setting the source IP address to an IPv4 address, the network traffic domain is obtained. ,in , ; Since multiple network flow packets may belong to the same network flow, each user... Network flow size is denoted as The cardinality of network flow is denoted as The upper bound of the network flow size is denoted as Correspondingly, each user The collected network stream data can be converted into a vector, denoted as ; After being added to the corresponding group, each user A counter can be used to count ; To reduce storage overhead, users use Hyperloglog Sketch for estimation. .

3. The top-based local differential privacy top-down method according to claim 1 The network flow estimation method is characterized by, Step S3 includes the following process: Each group Each user Network flow data collected over a period of time From the original length is domain hash to Store in a long sketch; Each location in each user's sketch The value stored in is calculated as follows: network stream data first passes through... Hash function hash to position Then through The hash function hashes the network stream to a value. The result of summation; Finally, privacy protection is implemented for the data in Sketch by adding... The noise is perturbed to satisfy the defined Adjacent - Local differential privacy model.

4. The top-based local differential privacy top-down method according to claim 1 The network flow estimation method is characterized by, Step S4 includes the following process: The server provides each group Enumerate fixed length prefix set The server is pre-configured , ; When estimating the frequency of each prefix in the network flow prefix set for each packet, the steps are as follows: ， The server will start from the first group Begin by generating top- Set of network flow prefixes ; Then, the server uses the existing top- The set of network flow prefixes is expanded to estimate the next group top- The number of times all possible extensions of the network flow prefix occur; Repeat the previous step until the total length of the constraint is reached, and finally obtain the estimated top-order value through iteration. Network flow.

Images