A distributed privacy-preserving record linking method

Abstract

This invention discloses a distributed privacy-preserving record linking method. The method utilizes the Apache Spark Standalone architecture and compute nodes to build a multi-threaded computing engine. A master server distributes stream data to worker nodes based on load balancing principles. Worker nodes receive multiple stream data to be matched and tokenize and hash the stream data field values ​​using the Mobile Distance Function (MBF) filter. Matching is performed using an improved metric distance function filtering method based on pivots. The master server allocates computing tasks based on the occupancy of each worker node. Once the link is established, a trusted link unit returns all successfully matched stream data to both data holders. This method uses pivots to limit the size of the similarity search space, reducing unnecessary similarity matches and significantly improving linking speed. The MBF not only improves security against frequency analysis attacks but also enhances storage efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of privacy protection technology, and in particular is a distributed privacy protection record linking method. Background Art

[0002] Privacy Preserving Record Linkage (PPRL) aims to link records related to individuals from different data sources while preserving privacy. Record linkage or entity resolution aims to link records that refer to the same real-world entity, such as a person or a product. Global identifiers are often lacking, so linkage can only be achieved by comparing available quasi-identifiers, such as name, address, or date of birth. However, in many cases, data owners are willing or allow their data to be provided for such data integration only if sensitive information is adequately protected to ensure the privacy of individuals such as patients or customers.

[0003] PPRL is a promising approach that allows the integration and use of data related to people from different sites without revealing the identity of the person. For this purpose, the linkage of records related to people is based on the coded values ​​of quasi-identifiers (QIs), and the data required for analysis (e.g., health data) is separated from these QIs. PPRL of two data sources (e.g., hospitals) and the linking is performed using a trusted unit. This linking unit only receives the coded QIs and not the health data relevant for analysis and medical research. The approach allows the combination of medical data from different sites to improve data analysis and research. These PPRL methods can be applied in many fields, such as public health monitoring, crime detection, demographic research, and market analysis.

[0004] PPRL faces many challenges that need to be addressed to ensure its practical applicability. In particular, a high degree of privacy must be ensured by appropriately encoding sensitive data and organizational structures, such as using trusted linking units. Despite the use of encoded QIs, PPRL must achieve high link quality by avoiding false or missing matches. In addition, high efficiency with fast connection time and scalability to large data volumes is required. A major performance issue is the quadratic complexity inherent in the linking problem when each record of the first source is compared with each record of the second source. To improve efficiency, the number of comparisons can be reduced by adopting blocking or filtering methods. In addition, the linkage unit can execute PPRL in parallel on multiple processing nodes.

[0005] Applications of PPRL are often found in crime and fraud detection, government services, and healthcare. PPRL can also be adapted to perform privacy-preserving similarity search (PPSS). In this scenario, an entity is no longer matched to a single other entity, but rather to multiple similar entities. For example, this can be used to find similar patients based on their medical records. Other possible applications of PPSS include clinical trial analysis and automated personalized healthcare software for specific patient populations.

[0006] PPRL is characterized by several classes of methods. One class is based on secure multi-party computation (SMC), which utilizes cryptographically strong protocols. These protocols enable similarity comparisons to be performed without revealing any information except the inputs and outputs of the protocol. While secure in principle, the computationally expensive cryptographic techniques required to implement SMC do not scale well to very large databases. This is because record-link methods are fundamentally statistical procedures that require model development and subsequent evaluation of model fitness.

[0007] As an alternative, the second type of PPRL exploits a weaker form of security based on data transformation. These transformed values ​​are called encodings and are used as input to PPRL. When determining a suitable data transformation method for PPRL, it is crucial to balance the three competing priorities of accuracy, security, and efficiency. A recent evaluation showed that transformations based on encoded values ​​in Bloom filters (BFs) outperform other methods (e.g., unique hashing or fuzzy identifiers, SMC-based equijoins, and SMC-based similarity comparisons) in terms of efficiency and accuracy. This approach, called field-level Bloom filter (FBF) encoding, maps each field value in a record to a corresponding Bloom filter.

[0008] However, current PPRL methods still have some shortcomings. Regarding security, Elizabeth ADurham et al. found that ordinary Bloom filters are vulnerable to frequency attacks that can reveal the stored values. Regarding efficiency, traditional privacy-preserving record linking methods are extremely inefficient when faced with large amounts of data that need to be linked. Summary of the Invention

[0009] The present invention aims to solve the technical problem of linking millions of records with privacy protection, which requires not only improving the linking speed but also resisting frequency attacks to enhance the security of the linked records. A distributed privacy protection record linking method is proposed.

[0010] In a first aspect, the present invention provides a privacy-preserving record linking method, comprising:

[0011] Build a distributed framework; the distributed framework uses the Apache Spark Standalone architecture combined with computing nodes in a multi-threaded manner, including the master server Cluster Manager and worker nodes Worker Node; Spark is a big data parallel computing framework based on in-memory computing, and the Standalone cluster uses the master-worker model in distributed computing. The master is the node in the cluster containing the master process, and the worker is the worker node in the cluster containing the Executor process.

[0012] The main server Cluster Manager obtains multiple stream data to be matched and distributes the above stream data to each worker node according to the load balancing principle;

[0013] The WorkerNode receives multiple stream data to be matched, and uses the Multi-Dimensional Bloom Filter (MBF) to tokenize and hash the stream data field values ​​based on statistical information.

[0014] Based on the improved metric distance function filtering method of the pivot, the worker node WorkerNode matches the stream data processed by the multi-dimensional Bloom filter;

[0015] The master server allocates computing tasks based on the occupancy of each worker node, and performs efficient and parallel privacy-preserving record linking.

[0016] After the link is completed, the trusted link unit will return all the successfully matched stream data information to both data holders, and the data holder can obtain the part of the stream data that successfully matches the stream data it holds.

[0017] Based on the Spark framework, a composite bloom filter is used to convert link data into a Bloom filter. This approach increases the randomness and anti-interference properties of the bloom filter, reducing the likelihood of successful frequency attacks against it. This method leverages the advantages of RBF encoding, such as high accuracy and fast runtime, while also increasing resistance to frequency-based cryptanalysis. Composite RBF encoding provides stronger resistance to frequency analysis and robustness, thus offering enhanced security.

[0018] Using a pivot-based filtering method based on a metric distance function saves many similarity calculations, further speeding up join time. The effectiveness of using pivots depends heavily on the number of pivots and how they are chosen. Generally, only heuristics can find good pivots. A minimum overlap between pivot radii is considered a necessary requirement, as it reduces the need to compare multiple records with multiple pivots and their assigned elements. This suggests that pivots on the "edge" of the metric space are desirable.

[0019] In a second aspect, the present invention provides a privacy-preserving record linking system for implementing the above method, comprising:

[0020] The main server, the Cluster Manager, is responsible for obtaining multiple stream data to be matched and distributing the stream data to each WorkerNode according to the load balancing principle. It also allocates computing tasks based on the occupancy of each WorkerNode, and performs efficient and parallel privacy-preserving record linking.

[0021] WorkerNode is responsible for receiving multiple stream data to be matched, tokenizing and hashing the stream data field values ​​using the Multi-Dimensional Bloom Filter (MBF) based on statistical information, and matching the stream data processed by the MBF using an improved metric distance function filtering method based on the pivot.

[0022] Trusted link unit, MBF returns all successfully matched flow data information to both data holders.

[0023] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described above.

[0024] In a fourth aspect, the present invention provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described is implemented.

[0025] Beneficial effects of the present invention:

[0026] This paper proposes a method for constructing a multidimensional Bloom filter using pivots and applying it to the Apache Spark architecture. The proposed multidimensional Bloom filter not only has a more complex storage structure but can also combine multiple sets of one-dimensional data into multidimensional data for storage, improving security against frequency analysis attacks and storage efficiency. By using pivots to limit the size of the similarity search space, unnecessary similarity matches are reduced, increasing the linking speed by approximately 5% compared to a method without pivots. The Apache Spark architecture enables analysis and linking of streaming data, breaking the limitation of traditional PPRL, which can only process batch data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the present invention.

[0028] Figure 2 This is the Apache Spark Standalone architecture diagram.

[0029] Figure 3 This is a flow chart of the pivot-based metric distance function filtering method.

[0030] Figure 4 is the relationship between link time and data volume with / without pivot at different matching rates.

[0031] Figure 5 Link time with / without pivot at different numbers of links.

[0032] Figure 6 The efficiency of the pivot is improved compared with that of the no pivot under different numbers of links. DETAILED DESCRIPTION

[0033] The following figure further illustrates the present invention in conjunction with the accompanying drawings and specific implementation steps:

[0034] This embodiment provides a privacy protection record linking method, including the following steps:

[0035] Step S1: Build a distributed framework as a computing engine; Figure 1 As shown, the data is divided into two parts, dataset R and dataset S, which are owned by data holder A and data holder B respectively.

[0036] The distributed framework adopts Figure 2 The Apache Spark Standalone architecture combines computing nodes in a multi-threaded manner. The master server Cluster Manager is responsible for scheduling and assigning tasks, and the worker node Worker Node is responsible for executing privacy-preserving record linking tasks.

[0037] The generated data is used as a dataset to build a computing engine for large-scale data processing based on the Apache Spark Standalone architecture, and interference is added to the data.

[0038] Step S2: The main server Cluster Manager obtains multiple stream data to be matched, and distributes the above stream data to each worker node according to the load balancing principle;

[0039] Step S3: The WorkerNode receives multiple stream data to be matched and tokenizes and hashes the stream data field values ​​using a multidimensional Bloom filter based on statistical information. Specifically, 3-1 first creates multiple bit vectors of length m, divides the stream data into units of m, and fills each bit vector into a regular Bloom filter to form a preliminary multidimensional Bloom filter (PMBF).

[0040] 3-2 Customize the weight and extraction order of the extracted fields, extract n bits from the m bits of the preliminary multi-dimensional Bloom filter (PMBF) to form a complex preliminary multi-dimensional Bloom filter (CPMBF), n<m;

[0041] 3-3 Generate a random sort N according to the custom parameters, sort the elements in the complex preliminary multi-dimensional Bloom filter (CPMBF) according to the random sort N, form a multi-dimensional Bloom filter (MBF), and obtain the tokenized and hashed field values ​​of the stream data.

[0042] Step S4: Based on the improved metric distance function filtering method of the pivot, the worker node WorkerNode matches the stream data processed by the multi-dimensional Bloom filter, see Figure 3 , specifically:

[0043] 4-1 Use MapPartition to determine the local pivots on each partition.

[0044] 4-2 Determine Global Pivots:

[0045] The stream data processed by the composite Bloom filter is partitioned using the GroupReduce operator, where each partition is configured with a local pivot. A global pivot is selected from the local pivots of each partition and stored in a distributed cache, which can be accessed on each partition.

[0046] 4-3 Index data holds the two-way flow data and assigns it to the local pivot closest to the global pivot, and then obtains the Hamming distance between each flow data and the local pivot.

[0047] The Hamming distance calculation formula is as follows:

[0048] d(p a ,q b )=|p a ∨q b |-|p a ∧q b Formula (1)

[0049] where d(p a ,q b ) is the local pivot p a With streaming data q b The Hamming distance of .

[0050] 4-4 Group the stream data of each local pivot through the GroupReduce operator to further determine the query radius of each local pivot;

[0051] By querying the radius rad(q b ) and combined with inequality (2), we can exclude dissimilar objects:

[0052] d(p a ,q b )-d(p a ,i c )≤rad(q b ) Formula (2)

[0053] Among them, d(p a ,q b ) is the local pivot p a To a single stream data q b The Hamming distance, d(p a ,i c ) is the local pivot p a To a single stream data i c Hamming distance of the stream data q b 、i c are the data holding the flow data of both parties, rad(q b ) is a single stream data q b The query radius.

[0054] 4-5 The data holders transfer the flow data processed by the multi-dimensional Bloom filter to the trusted link unit for link calculation: the flow data of the data holders are traversed in turn to find the matching flow data of the data holders; specifically:

[0055] 4-5-1 Initialization: a=1, b=1, c=1, iteration number t=1;

[0056] 4-5-2 Due to the first data holder's stream data q b Determine the global pivot that may contain matching candidates and judge the flow data q b With local pivot p a Is the distance less than q b When the query radius is , if it is considered that p a With q b If there is a certain similarity in the data structure, execute step 4-5-3; if not, determine whether a is less than A, A represents the total number of local pivots, if so, update a=a+1 and repeat step 4-5-2, if not, determine whether b is less than B, B represents the total number of stream data of the first data holder, if so, update b=b+1 and repeat step 4-5-2, if not, end;

[0057] 4-5-3 for each flow data q b Determine the global pivots that may contain matching candidates.

[0058] Determine q using the FlatMap operator according to formula (3) b Similarity radius rad(q b ):

[0059]

[0060] According to formula (4), the local pivot p is determined a To stream data q b The Hamming distance d(p a ,q b ) is greater than the local pivot p a and stream data q b The sum of the query radius, if it is greater than, it is considered as streaming data q b Does not belong to the local pivot p a , that is, the data holding the two stream data do not match, continue to determine whether b is less than B, if so, update b = b + 1, repeat steps 4-5-2, if not, end;

[0061] If it is less than or equal to the flow data q b belongs to the local pivot p a , and then judge the flow data q according to formula (5) b and i c Do they all belong to the local pivot p? a If so, then the stream data q b and i cMatch, otherwise it does not match, continue to determine whether c is less than C, C represents the total number of stream data of the second data holder, if so, update c = c + 1 and repeat steps 4-5-3, if not, end;

[0062] d(p a ,q b )>rad(q b )+rad(p a )

[0063] Formula (4)

[0064] d(p a ,q b )-d(p a ,i c )>rad(q b )

[0065] Step S5 of formula (5): The master server allocates computing tasks according to the occupancy of each working node and performs efficient and parallel privacy-preserving record linking.

[0066] Step S6: After the link is completed, the trusted link unit returns all successfully matched stream data information to both data holders, and the data holder can obtain the part of the stream data that successfully matches the stream data it holds.

[0067] The following link experiments were conducted for different link data sizes and different matching rates. The experimental data are as follows:

[0068] Table 1 Linking time of the present invention with different link data amounts and different matching rates (with pivot)

[0069]

[0070] Table 2 Linking time consumption (without pivot) of the method without step S4 for different link data amounts and different matching rates

[0071]

[0072] Table 3 Rates of the method of the present invention for different link data amounts and different matching rates

[0073]

[0074] Depend on Figure 4 (a)-(b) shows that different matching rates have no significant effect on link time, and the size of the data is the main reason affecting the link time. Figure 5 It can be seen that as the number of connections increases, the link time with pivots decreases more significantly than the link time without pivots. Figure 6It can be seen that regardless of the data volume, the connection speed of the pivot always maintains a lead of about 5% over the non-pivot. When the data volume increases, even if the improved efficiency remains at 5%, the time reduction brought by the pivot will become more obvious as the overall connection time increases.

Claims

1. A privacy protection record linking method, characterized in that The method comprises: Build a distributed framework; the distributed framework uses the Apache Spark Standalone architecture combined with computing nodes in a multi-threaded manner, including the main server Cluster Manager and worker nodes; The main server Cluster Manager obtains multiple stream data to be matched and distributes the above stream data to each worker node according to the load balancing principle; The Worker Node receives multiple stream data to be matched and tokenizes and hashes the stream data field values ​​using a multi-dimensional Bloom filter (MBF) based on statistical information. Based on the improved metric distance function filtering method of the pivot, the worker node matches the stream data processed by the multi-dimensional Bloom filter; The master server allocates computing tasks based on the occupancy of each worker node, and performs efficient and parallel privacy-preserving record linking. After the link is completed, the trusted link unit will return all the successfully matched stream data information to both data holders, and the data holder can obtain the part of the stream data that successfully matches the stream data it holds.

2. The method according to claim 1, characterized in that The worker node receives multiple stream data to be matched, and tokenizes and hashes the stream data field values ​​using a multi-dimensional bloom filter (MBF) based on statistical information, specifically: First, multiple bit vectors of length m are created, the stream data is divided into units of m, and each bit vector is filled into a common Bloom filter to form a preliminary multidimensional Bloom filter PMBF. Customize the weight and extraction order of the extracted fields, extract n bits from the m bits of the preliminary multidimensional Bloom filter PMBF, and form a complex preliminary multidimensional Bloom filter CPMBF, where n < m; Generate a random sort N according to the custom parameters, sort the elements in the complex preliminary multi-dimensional Bloom filter CPMBF according to the random sort N, form a multi-dimensional Bloom filter MBF, and obtain the tokenized and hashed field values ​​of the stream data.

3. The method according to claim 1, characterized in that In the pivot-based improved metric distance function filtering method, the worker node matches the stream data processed by the multi-dimensional Bloom filter, specifically: 1) Use MapPartition to determine the local pivots on each partition; 2) Determine the global pivots; 3) The index data holds the two-way flow data and assigns it to the local pivot closest to the global pivot, and then obtains the Hamming distance between each flow data and the local pivot; 4) The stream data of each local pivot is grouped by the GroupReduce operator to further determine the query radius of each local pivot; by b ) and exclude dissimilar objects by combining inequality (2): Among them, d(p a ,q b ) is the local pivot p a To a single stream data q b The Hamming distance, d(p a ,i c ) is the local pivot p a To a single stream data i c Hamming distance of the stream data q b 、i c are the flow data of both parties, rad(q b ) is a single stream data q b The query radius; 5) The data holders transfer the flow data processed by the multi-dimensional Bloom filter to the trusted link unit for link calculation: traverse the flow data of the data holders in turn to find the matching flow data of the data holders.

4. The method according to claim 3, characterized in that The determination of the global pivots is specifically to use the GroupReduce operator to partition the stream data after the composite Bloom filter is processed, where each partition is configured with a local pivot; the global pivot is selected from the local pivots of each partition, and the above global pivots are stored in a distributed cache and can be accessed on each partition.

5. The method according to claim 3, characterized in that The Hamming distance calculation formula is as follows: where d(p a ,q b ) is the local pivot p a With streaming data q b The Hamming distance of .

6. The method according to claim 3, characterized in that The data holders hand over the flow data processed by the multi-dimensional Bloom filter to the trusted link unit for link calculation: the flow data of the data holders are traversed in sequence to find the matching flow data of the data holders, specifically: 5-1 Initialization: a=1, b=1, c=1, iteration number t=1; 5-2 Due to the first data holder's stream data q b Determine the global pivot that may contain matching candidates and judge the flow data q b With local pivot p a Is the distance less than q b When the query radius is , if it is considered that p a With q b If the data structure has a certain similarity, execute step 5-3; if not, determine whether a is less than A, where A represents the total number of local pivots. If so, update a=a+1 and repeat step 5-2. If not, determine whether b is less than B, where B represents the total number of stream data of the first data holder. If so, update b=b+1 and repeat step 5-2. If not, end; 5-3 is for each flow data q b Determine global pivots that may contain matching candidates; Determine q using the FlatMap operator according to formula (3) b Similarity radius rad(q b ): According to formula (4), the local pivot p is determined a To stream data q b The Hamming distance d(p a ,q b ) is greater than the local pivot p a and stream data q b The sum of the query radius, if it is greater than, it is considered as streaming data q b Does not belong to the local pivot p a , that is, the data holding the two stream data do not match, continue to determine whether b is less than B, if so, update b = b + 1, repeat step 5-2, if not, end; If it is less than or equal to the flow data q b belongs to the local pivot p a , and then judge the flow data q according to formula (5) b and i c Do they all belong to the local pivot p? a If so, then the stream data q b and i c Match, otherwise it does not match, continue to determine whether c is less than C, C represents the total number of stream data of the second data holder, if so, update c = c + 1 and repeat step 5-3, if not, end; d(p a ,q b )>rad(q b )+rad(p a ) Formula (4) d(p a ,q b )-d(p a ,i c )>rad(q b ) Equation (5).

7. A privacy protection record linking system implementing the method according to any one of claims 1 to 6, characterized in that include: The main server, the Cluster Manager, is responsible for obtaining multiple stream data to be matched and distributing the stream data to each WorkerNode according to the load balancing principle. It also allocates computing tasks based on the occupancy of each WorkerNode, and performs efficient and parallel privacy-preserving record linking. WorkerNode is responsible for receiving multiple stream data to be matched, tokenizing and hashing the stream data field values ​​using the Multi-Dimensional Bloom Filter (MBF) based on statistical information, and matching the stream data processed by the MBF using an improved metric distance function filtering method based on the pivot. Trusted link unit, MBF returns all successfully matched flow data information to both data holders.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 6.

9. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.

Images