A method and system for finding two-party privacy intersections that supports arbitrary data sizes

Abstract

This invention discloses a method and system for privacy-preserving intersection of two parties supporting arbitrary data scales, relating to the field of data security processing technology. The method includes the following steps: obtaining the original data sets of the two parties requiring interaction; binning the original data of both parties; performing deduplication in parallel on each bin; generating a corresponding transparent vector linear estimate quantity by performing security protection operations on a single bin based on the amount of data in the initiator's bin; performing privacy-preserving intersection of the data by the initiator and receiver using unintentional key-value storage based on the transparent vector linear estimate quantity; comparing, sorting, and searching the deduplicated data within each bin to obtain the position of the intersection element in the original datasets of both parties before deduplication. This invention achieves fast privacy-preserving intersection in data scenarios of arbitrary scales, providing a complete and efficient solution for the widespread application of privacy-preserving intersection in practical business applications.

Description

Technical Field

[0001] This invention relates to the field of data security processing technology, and in particular to a method and system for finding privacy-preserving intersections of arbitrary data sizes. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous development of the internet and information technology, the collection and utilization of user data is showing an increasing trend. Against this backdrop, effectively protecting user privacy has become a key issue in information system design. In many application scenarios, different entities (such as organizations or individuals) need to maintain the privacy of their respective data when conducting joint analysis or data exchange. However, given that shared data may contain sensitive information, how to achieve data collaboration without leaking the original data content poses a significant challenge to existing data interaction technologies.

[0004] Privacy-preserving intersection refers to finding the intersection of elements in both sets without exposing the original data. Currently, the mainstream privacy-preserving intersection protocols fall into two main categories: the ECDH scheme based on elliptic curve operations and the KKRT scheme based on unintentional transport protocols (OT). However, both have inherent drawbacks. The ECDH scheme has better communication complexity, but the complex elliptic curve operations lead to expensive computational complexity. The KKRT scheme has better computational complexity, but the OT protocol introduces expensive communication complexity. Later, the RR22 scheme solved the balance problem between communication and computational complexity. This scheme proposes an efficient linear unintentional key-value store (OKVS) construction and uses Transparent Vector Linear Estimation (VOLE) to protect the encoded vectors. The combination of these two methods constructs an unintentional pseudo-random function (OPRF) to achieve privacy-preserving set intersection, making it the PSI scheme with the best computational and communication complexity to date.

[0005] However, due to the limitations of the algorithm itself, the RR22 scheme has the following drawbacks in actual development:

[0006] 1. When dealing with small datasets, the solution is unsuitable for a few or dozens of data entries because VOLE's extended OT protocol requires a minimum of 128 calls to the basic OT protocol, and OKVS uses the AVX parallel instruction set. 2. OKVS requires unique keys and values, necessitating an efficient deduplication method while also preserving the location of the original dataset. 3. When dealing with large datasets, such as billions or tens of billions of records, the memory-intensive algorithms are not suitable for many real-world scenarios, requiring modifications and adaptations to the algorithm and protocol flow.

[0007] In conclusion, how to support data scenarios of any type and scale while maintaining high computational efficiency during two-party privacy intersection processes has become an urgent problem to be solved by existing technologies. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for two-party privacy intersection that supports arbitrary data scales. This overcomes the deficiencies of existing RR22 schemes in privacy intersection, enables fast privacy intersection in data scenarios of arbitrary scales, and provides a complete and efficient solution for the widespread application of privacy set intersection in practical business.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0010] The first aspect of this invention provides a two-party privacy intersection method supporting arbitrary data size, comprising the following steps:

[0011] Obtain the original data sets of the two parties that need to interact, determine the initiator and receiver based on the data volume, and bucket the original data of both parties respectively;

[0012] Deduplication is performed in parallel for each bucket, and the deduplicated data is retained.

[0013] Based on the amount of data in a single bucket after the initiator's bucketing, generate the corresponding transparent vector linear estimate of the quantity for both parties;

[0014] Based on the transparent vector linear estimation quantity, the initiator and receiver use unintentional key-value storage to perform privacy intersection on the data;

[0015] The deduplicated data within each bucket is compared, sorted, and searched to obtain the position of the intersection element in the original datasets of both sides before deduplication.

[0016] A second aspect of the present invention provides a two-party privacy intersection system supporting arbitrary data size, comprising:

[0017] The bucketing module is configured to obtain the original data sets of the two parties that need to interact, determine the initiator and receiver based on the data volume, and bucket the original data of both parties respectively.

[0018] The deduplication module is configured to perform deduplication in parallel for each bucket, retaining the deduplicated data;

[0019] The intersection relationship establishment module is configured to perform security protection operations on a single bucket based on the amount of data possessed by a single bucket after the initiator buckets the data, generating a corresponding transparent vector linear estimate of the number of data.

[0020] The index mapping module is configured to estimate the quantity based on a transparent vector linear estimation, and the initiator and receiver use an unintentional key-value store to perform privacy intersection on the data.

[0021] The adjustment and recovery module is configured to compare, sort, and search the deduplicated data within each bucket to obtain the position of the intersection element in the original datasets of both parties before deduplication.

[0022] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute steps in the two-party privacy intersection method supporting arbitrary data size as described in the first aspect of the present invention.

[0023] A fourth aspect of the present invention provides a computer device comprising:

[0024] A processor, adapted to execute computer programs;

[0025] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the two-party privacy intersection method supporting arbitrary data size as described in the first aspect of the present invention.

[0026] The above one or more technical solutions have the following beneficial effects:

[0027] This invention discloses a method and system for two-party privacy intersection that supports arbitrary data scales, applicable to large-scale data matching scenarios such as encrypted communication and Private Set Intersection (PSI). This method improves upon the existing RR22 scheme, utilizing a strategy combining bucketing parallelism and MMAP to achieve fast privacy intersection for data scenarios of arbitrary scales, providing a complete and efficient solution for the widespread application of privacy set intersection in practical business. First, an improved hash structure is used to establish a fast index mapping table for one set of data. Then, the intersection relationship with the other set is efficiently obtained. The final output array stores the indices of matching elements in one set in the first half and the corresponding indices of matching elements in the other set in the second half, thus achieving a bidirectional index mapping of intersection elements. This invention not only avoids the high time complexity of traditional sorting-merge methods but also significantly improves the overall efficiency of set intersection operations through hash structures and parallel mechanisms, making it suitable for deployment in high-performance privacy computing systems. This invention uses open-source radix sort to sort the positional results. Finally, all matching results are linearly stored in a contiguous block of memory and returned to the caller. The method of this invention supports features such as duplicate value matching, order preservation, and multi-target merging output, which can efficiently complete the typical requirement of "multi-key lookup" while maintaining excellent performance in terms of space and time overhead.

[0028] This invention determines the number of intersection elements through hash lookup, thereby accurately allocating storage space in advance; then, it uses parallel technology (OpenMP) to perform concurrent searches on the set, and uses atomic counters to write the matching results while ensuring thread safety.

[0029] This invention proposes a fast reverse index lookup method for discrete numerical sets, particularly suitable for scenarios where the target value set is small and the original array is large, such as data filtering, encrypted matching, and privacy queries. First, a mapping relationship is constructed between each value in the input array and all its occurrence positions. A data structure is used to ensure that both insertion and lookup operations have near-constant time complexity. After construction, the method extracts all indices of the target values ​​in the original array according to the specified target values ​​and outputs them in a stable order.

[0030] This invention first adapts VOLE and OKVS using bucketing and MMAP technologies, supporting data volumes of any scale and providing a reference for data volumes of billions and tens of billions in practical engineering. Secondly, it provides efficient deduplication, padding, comparison, sorting, and search functions, offering a series of auxiliary means for privacy-preserving intersection calculations in real-world applications. Finally, the maximum number of parallel operations supported between buckets and the method for combining VOLE and OKVS provide optimization techniques to further reduce runtime memory peaks.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of the two-party privacy intersection method supporting arbitrary data size in Embodiment 1 of the present invention;

[0034] Figure 2 This is a schematic diagram of the deduplication step in Embodiment 1 of the present invention;

[0035] Figure 3 This is a schematic diagram of MMAP mapping in Embodiment 1 of the present invention. Detailed Implementation

[0036] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0037] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0039] Example 1:

[0040] Embodiment 1 of this invention provides a two-party privacy intersection method that supports arbitrary data sizes, improving upon the existing RR22 scheme, such as... Figure 1 As shown, it includes the following steps:

[0041] Step 1: Obtain the original data sets of the two parties that need to interact, determine the initiator and receiver based on the data volume, and bucket the original data of both parties respectively.

[0042] In one specific implementation, the initiator and receiver are determined based on the amount of data. Specifically, in a balanced scenario (where both sides have roughly equal amounts of data), either side acts as the initiator, and the other as the receiver. If the difference in data volume between the two sides exceeds a set threshold, i.e., in an unbalanced scenario, the side with the smaller amount of data becomes the initiator, and the other becomes the receiver. In this embodiment, an unbalanced scenario is defined as a significant difference in data volume between the two sides, such as 100,000 versus 100 million or 1 million versus 100 million, with a difference of at least two orders of magnitude. Taking the intersection of 128-bit block data sets as an example, this embodiment uses an improved hash structure `robin_hood::unordered_flat_map` to establish a fast index mapping table for one side's data set, where each 128-bit block is bound to its position in the set. In this embodiment, one side's data set is denoted as set X, and the other side's data set is denoted as set Y. Specifically, the original dataset is stored in a CSV file. The file is read using the Go language's official standard library encoding / csv. Each time r.Read() is called, the underlying system reads a block of data from the file system into a memory buffer, and then parses the CSV from the buffer to read the data line by line. This method has low memory usage and is suitable for processing large files. The reading speed depends on I / O.

[0043] It is important to note that the data from both sides must be bucketed in the same way, i.e., using the same hash function and modulo operation. This ensures that the intersection element is in the corresponding bucket. It does not matter whether the amount of data in the corresponding buckets is the same for both sides after bucketing. It is only necessary to use the modulo operation to ensure that the range of data in the corresponding buckets is consistent for both sides after bucketing.

[0044] The binning process is shown in the attached document. Figure 1 As shown, the specific steps are as follows:

[0045] Step 1.1: The data in the CSV file is read into memory one row at a time.

[0046] Step 1.2: Calculate a 128-bit hash value for a single data entry using the SM3 hash function.

[0047] Step 1.3: First set There are n buckets, where n is the specified number of buckets. The first n buckets store the hash result of the original data, and the last n buckets store the original index of the original data before bucketing. The lower 8 bytes of the hash value calculated in step 1.2 are concatenated in big-endian format, as shown below:

[0048] hash64=hash[0]<<56|hash[1]<<48|hash[2]<<40|hash[3]<<32|hash[4]<<24|hash[5]<<16|hash[6]<<8|hash[7].

[0049] Then, calculate pos = hash64%n, where pos represents the bucket number the current data should be assigned to. Simultaneously, save the current data's position in the original set to the (pos+n)th bucket. To reduce memory usage, each bucket is an MMAP (Memory-Mapped File), meaning the calculated position and original index are written to the MMAP file. Therefore, this process requires significant memory consumption. One MMAP file.

[0050] Step 2: Perform deduplication in parallel for each bucket and retain the deduplicated data.

[0051] In one specific implementation, after the original data set undergoes the aforementioned bucketing operation, the intersection elements of the two sets are located in the corresponding buckets. Then, deduplication is performed on the data in each bucket. This embodiment provides an efficient deduplication and index mapping method for 128-bit vector data blocks. This method constructs a hash mapping structure with a dedicated hash function and equivalence comparison function for vector data using a `robin_hood` hash table (`robin_hood::unordered_map`), achieving fast deduplication of large amounts of vector data and mapping its position within the original data. The hash function decomposes the 128-bit vector into four 32-bit unsigned integers and constructs a composite hash value using a hybrid perturbation function. The equivalence comparison function is based on the Single Instruction Multiple Data (SIMD) instruction set, using bitwise XOR and all-zero detection to quickly compare whether two vector data sets are completely equal.

[0052] In hash design, a hybrid perturbation function refers to compressing high-dimensional or large-scale data into a hash value with good distribution characteristics by performing various mathematical transformations on the original data (such as multiplication perturbation, bitwise operations, shifting, XOR, etc.), thereby reducing the probability of hash collisions. Constructing composite hash values ​​using hybrid perturbation functions is a common technique in this field and will not be elaborated upon here.

[0053] The equivalence comparison function is used to determine whether two 128-bit SIMD vectors (i.e., block type, underlying __m128i) are completely identical. It uses SSE instructions to quickly perform byte-by-byte comparison, which is a very efficient way to determine vector equality, and will not be elaborated here.

[0054] During the process of traversing the input data, the algorithm determines in real time whether an element already exists in the unique set and simultaneously establishes the correspondence between the original index and the deduplicated set.

[0055] Specifically, as the algorithm iterates through each element in the input data, it checks in real-time in a unique set whether the element already exists. If the element "exists" in the unique set, it means it has already appeared in another set. In this case, the algorithm records both the element's index in the current set and its original index in the unique set, indicating that there is identical data in both sets, forming a one-to-one matching record. If the element "does not exist" in the unique set, it means it is a new value (not yet appeared in the other set). In this case, the current element will not be recorded in the result array, which is equivalent to discarding it (not participating in the intersection matching), and the algorithm continues to process the next element. Therefore, when an element "exists," an index mapping is established; when an element "does not exist," it is skipped without any operation.

[0056] As attached Figure 2 As shown, the deduplication function returns two arrays. The first array is the array of deduplicated blocks, and the second array is used for the final comparison and restoration of the original positions. Its size is the same as the original bucket data size. The array indices correspond to the indices of the original elements in the original data set, and each index stores the index of the deduplicated array element. The first deduplication array is used for subsequent filling and OKVS stages, while the second array is used to restore the positions of the original intersection data after the comparison stage. It is worth noting that the above operation is for deduplication of a single bucket, but the deduplication is performed in parallel between buckets. Therefore, in actual implementation, in addition to single-bucket deduplication, parallel deduplication of multiple buckets can also be implemented. Furthermore, the deduplication achieved by this algorithm is very fast in parallel and is also very efficient for subsequent restoration of the original positions.

[0057] Next, the data within each individual bucket is traversed. The traversal process includes an expansion phase, a VOLE phase, and an OKVS phase. First, one side of the data set is linearly scanned, and the number of intersection elements is determined by hash lookup, thus accurately allocating storage space in advance. Then, parallel technology (OpenMP) is used to perform concurrent searches on the data set, and the matching results are written using atomic counters while ensuring thread safety.

[0058] Step 3: Based on the amount of data in a single bucket after the initiator has created the bucket, perform security protection operations on the single bucket to generate the corresponding transparent vector linear estimate of the number of data.

[0059] In one specific implementation, some basic VOLE vectors, each 128 bits long, are first generated, with one side holding w and v, and the other side holding u and v. ,satisfy The linear relationship. The extended OT protocol is used in the process of generating the basic VOLE.

[0060] The basic VOLE vector is then extended. This embodiment uses a GGM Tree, an existing data structure whose data optimization process is common knowledge in the field and will not be elaborated here. The GGM Tree also uses an extended OT protocol in the construction and extension of the VOLE vector protocol. Therefore, the security protection operation in this embodiment essentially uses the extended OT protocol twice: once in the construction of the basic VOLE and once in the construction of the extended VOLE, but the initiators are different. In this embodiment, the extended OT protocol is optimized based on the basic OT protocol.

[0061] The expanded vector is One party holds A and C, and the other party holds And B. However, the vectors at this point do not satisfy the security problem in cryptography. Therefore, this embodiment constructs the LPN problem by expanding the convolutional code, specifically by multiplying the two vectors by the convolution matrix and the expansion matrix. The specific steps are as follows:

[0062] Step 3.1: Fill the data in the buckets whose data volume is less than the threshold.

[0063] When the amount of data in a single bucket is small, such as a few or dozens, the algorithm itself may cause some unexpected errors and fail to execute properly due to its limitations. This embodiment adopts the method of padding with random numbers. Random blocks are generated using AES ECB mode and appended sequentially to the end of the data in both buckets. When appending, it is not necessary to consider whether the generated random block has any intersection with the preceding elements or with the other party's elements, because after OKVS encoding and decoding, the appended random block will definitely be in the corresponding position of the padding set of both parties. The comparison only needs to compare the number of elements after deduplication. Therefore, padding only plays the role of ensuring that the algorithm can execute normally.

[0064] Step 3.2: Set the number of buckets using a combination of parallel and serial methods.

[0065] When dealing with large datasets, such as billions or tens of billions of data points, a number of VOLE operations at the billion or tens of billions level is required. However, general memory cannot support this. Therefore, it is necessary to set the number of parallel buckets, for example, first parallelizing 10 or 20 buckets, then parallelizing another 10 or 20 buckets, which combines serial and parallel processing, as shown in the attached diagram. Figure 3 The VOLE results for each bucket are concatenated in order of bucket number and written to the MMAP file to reduce memory usage.

[0066] This embodiment assumes the task initiator is the client, with the corresponding relationship being Client - Sender (S) - PA and Server - Receiver (R) - PB. First, the PA calculates the number of VOLEs to be generated based on its data volume and then sends it to the PB. The input to the VOLE only needs to know its generation count. The output of the VOLE is... Among them, A, B, C and All are VOLE vectors, representing VOLE A, VOLE B, VOLE C, and VOLE respectively. Vectors (VOLE delta vectors). Since subsequent steps involve the exchange of data between the sender and receiver, to avoid ambiguity, PA is defined as the original sender determined in step 1, holding VOLE C and VOLE A vectors; PB is the original receiver determined in step 1, holding VOLE B vector and... vector.

[0067] Step 3.3: Expand the data within a single bucket through the expansion phase.

[0068] Step 3.3.1: Basic OT Phase. The roles of both parties are reversed to perform basic OT, thus the PA becomes the receiver and the PB becomes the sender. This embodiment performs 128 basic OT operations, implemented through an internal loop. During execution, intermediate values ​​exchanged during the basic OT protocol process are sent together.

[0069] Step 3.3.1.1: PB requires no input; it internally generates a random number d and calculates the d-fold multiple of the base point g. The result is C( Send it to PA, and the function returns d and C.

[0070] Step 3.3.1.2: PA receives the received C and internally generates a random number seed (senderseed), thereby constructing a 128-bit selection vector s, and then generates a random number. The key vector sk is calculated if si=0. If si=1, then calculate Then the PA sends ( The function returns sk to PB. , senderseed, s. Where si is the selection bit vector. Intermediate values ​​to be sent during protocol execution

[0071] Step 3.3.1.3: The PB side inputs the received data. In addition to the previously generated C and d, which internally generate a random number seed `receiverseed`, and from this seed generate seeds T and U, using seeds T and U as the base OT. and Then you can calculate and = / Then calculate =Hash( )and =Hash( ), and then put and As a key pair and Encrypt to obtain ( )and ( The function sends the data to the PA and returns received, T, U, etc. , .

[0072] Step 3.3.1.4: PA inputs the received data. and In addition to the previously received C, the previously generated sk and s, then calculate =Hash(( ),use Decryption ( , This returns the seed Q. The function returns Q.

[0073] Step 3.3.2: Extended OT Phase. The roles of both parties are reversed, so PA becomes the sender again and PB becomes the receiver again.

[0074] Step 3.3.2.1: Input the previously generated received, T, and U into PB, and then randomly generate a 128-bit scalar internally. That is the final VOLE ▲. (By...) Construct the receiver selection bit vector r( ) and delta_bit (128bit). Then call the G function and pass in the random number seed T to calculate the T_M matrix. At the same time, call the G function and pass in the random number seed U to calculate the U_M matrix. Then fill the vector r' according to r. Then XOR U_M with matrix T_M and vector r' in turn to calculate the P matrix. Then P matrix ( The function sends the result to PA, then transposes matrix T_M and passes it to AES to calculate K. The function returns K, r, delta_bit, and P.

[0075] Step 3.3.2.2: PA takes the received P, the previously received Q, the previously generated senderseed, and s as input. It then calls the G function, passing in the random number seed Q. Based on the selected bit vector si (if 1), it calculates the result of XORing G(Q) with the received P matrix, which is the Q_M matrix. Next, it calculates the transpose QT of the Q_M matrix. Finally, it calls AES, passing in QT to calculate K0, and passing in the result of XORing QT with the selected bit vector si to calculate K1. The function returns K0 and K1.

[0076] Step 3.3.2.3: PA inputs the previously generated K0 and K1, then internally generates a random number seed_k, which in turn generates k0 and k1. Then, k0 and k1 are XORed with K0 and K1 respectively to obtain the result C0. C1 Send it to PB. The function returns k0, k1, C0, and C1.

[0077] Step 3.3.2.4: Input the previously generated r and K, as well as the received C0 and C1, into the PB side. Then, calculate the XOR result of K with C0 or C1 based on the randomly selected bit vector ri being 0 or 1. Finally, obtain the extended OT result, which is the desired k. The function returns k.

[0078] Step 3.4: Perform security protection operations on the expanded data through the VOLE stage to obtain the transparent vector linear estimate of the receiver.

[0079] Step 3.4.1: Basic VOLE Phase. Generate some basic VOLE vectors that satisfy... After the agreement is completed, PA holds u and w, and PB holds ▲ and v.

[0080] Step 3.4.1.1: The PA side inputs the previously generated k0, k1, and seed_k, uses k0 and k1 as plaintext, and calls AES encryption to generate w0 and w1. The PA side randomly generates the u vector and calculates gama = w0⊕w1⊕u( The function sends the result to PB. Finally, PA calculates the inner product vector of w0 on GF128 as the final w, and the function returns u, w, and gama.

[0081] Step 3.4.1.2: The PB side inputs the previously obtained k obtained through extended OT, the previously generated delta_bit, and the recently received gama. The received k is used as plaintext, and AES encryption is called to generate w.

[0082] Then, based on delta_bit, v = w⊕▲·gama is calculated. The function returns v.

[0083] After the above basic VOLE procedure is completed, PA obtains u and w, and PB obtains v, satisfying the condition. .

[0084] Step 3.4.2: Extended VOLE Stage. The basic VOLE vector is extended and an LPN problem is constructed using GGM Tree, Exconvcode, etc., ultimately achieving... The effect is that PA holds A and C, and PB holds B and ▲.

[0085] Step 3.4.2.1: The PB side inputs the length of the OKVS encoding vector and the previously generated v. Internally, it first randomly samples a seed_k', then calls GGM.FullEval() to construct the GGM Tree, and finally calculates the XOR sum of all leaf nodes in the GGM Tree, as well as the XOR sum m0' of the left subtree nodes and the XOR sum m1' of the right subtree nodes at each level. Then, it XORs the XOR sum of all leaf nodes with the v they hold to obtain sendtoR( Send to PA. The function returns seed_k', B, m0', m1', and sendtoR.

[0086] Step 3.4.2.2: PB receives seed_k' and B, generates ec_key (128 bits) based on seed_k', sends it to PA, and then calls the extended convolutional code function to extend B. The function returns ec_key.

[0087] At this point, PB has obtained the final B. All that remains is for PA to obtain A and C.

[0088] Step 3.5: Repeat the expansion and security protection operations to obtain the linear estimate of the initiator's transparent vector. That is, this step obtains the PA's VOLE C vector and VOLE A vector.

[0089] PB needs to send m0 and m1 to PA through extended OT, so the initial extended OT process is repeated.

[0090] Step 3.5.1: Repeat the basic OT phase.

[0091] Step 3.5.1.1: PA has no input, so C'( Send it to PB, and the function returns d' and C'.

[0092] Step 3.5.1.2: PB inputs C' and sends. ( The function returns sk' to PA. 、senderseed、s'.

[0093] Step 3.5.1.3: PA inputs C', d', ,get ( )and ( Send it to PB, and the function returns received', T', U', Y0', Y1'.

[0094] Step 3.5.1.4: Input C', sk', Y0', Y1', and s' into PB to obtain Q', and the function returns Q'.

[0095] Step 3.5.2: Repeat the extended OT phase.

[0096] Specifically, a 128-bit basic VOLE vector is first constructed by calling the extended OT protocol (basic OT). One side holds w and v, and the other side holds... and u, satisfying the relation Then, the basic VOLE vectors are expanded by constructing a GGM Tree (which generates a large number of linearly dependent VOLE vectors with fewer communication rounds) to meet the final data volume requirements. During this process, an extended OT protocol is invoked to transmit intermediate values. In the extended VOLE generation, one party holds A and C, and the other party holds... And B, satisfy However, since the vectors at this point do not meet the security requirements in cryptography, the extended convolutional code is called to construct the LPN problem, that is, each vector is multiplied by a convolution matrix and an extension matrix, and finally a VOLE vector that meets the requirements is generated for use in the OKVS stage.

[0097] Step 3.5.2.1: PA is input with received', T', and U'. Then, through division, modulo operations (GenRandomMod()), t(128) piercing positions are randomly sampled and converted into a selection bit vector r''' (the length of the selection bit is determined by the depth of the tree structure, O(logn)). This vector is then input into the previously extended OT function to obtain P_New( Send to PB, the function returns last_len, level, level_last1, sub_len, index, K', P_New and r'''.

[0098] Step 3.5.2.2: Input the received PNew, senderseed', s', and Q' into the PB side, and the function returns K0' and K1'.

[0099] Step 3.5.2.3: PB inputs the previously generated m0', m1' and the newly generated K0', K1', and performs an XOR operation to obtain... and Send to PA. The function returns C0' and C1'.

[0100] Step 3.5.2.4: PA inputs r''', K', C0', C1' to obtain the final m'.

[0101] Where m' is either m0' or m1' selected through extended OT.

[0102] The function returns m'.

[0103] Step 3.5.3: Extend VOLE in the final stage.

[0104] The PA function takes ec_key, last_len, level, level_last1, sub_len, index, sendtoR, m', u, w, r''' as input and obtains the final C and A. The function returns C and A.

[0105] In the above process, the final generated VOLE A and VOLE C vectors are written to the MMAP file of the PA side, and the VOLE B vector is written to the MMAP file of the PB side, while saving 128 bits. value.

[0106] Step 4: Based on the linear estimation of the quantity using transparent vectors, the initiator and receiver use unintentional key-value storage to perform privacy-preserving intersection of the data.

[0107] In one specific implementation, this step is the OKVS phase, which performs a privacy intersection on the data of both parties based on security protection. The initiator encodes its data using Unintentional Key-Value Storage (OKVS) and masks the encoded result P with the A vector of VOLE (i.e., A+P) before sending it to the receiver. Then, the initiator uses the C vector of VOLE as the OKVS result and inputs it into its own data for decoding to obtain the final encrypted result. After receiving A+P, the receiver executes... ,get Then The OKVS result is input into the receiver's data for decoding, and the decoded result is then XORed again. If there is an intersection between the two sets, both sets will decode C, resulting in the same outcome while simultaneously concealing the original information. The first half of the final output array in the OKVS phase stores the indices of matching elements in set Y, and the second half stores the indices of corresponding matching elements in set X, thus achieving a bidirectional index mapping of the intersection elements. This method not only avoids the high time complexity of traditional sorting-merge methods but also significantly improves the overall efficiency of set intersection operations through hash structures and parallel mechanisms, making it suitable for deployment in high-performance privacy computing systems. The maximum number of parallel buckets for OKVS encoding and decoding of data in each bucket is consistent with the VOLE phase. It is only necessary to calculate the starting address and offset of MMAP reasonably according to the required number of VOLE operations for each bucket, and then extract the vectors A, B, and C. The specific steps are as follows:

[0108] Step 4.1: PA inputs its own data set X and the A it holds, and internally first generates a random number seed r_new ( r_new is used for encoding. Then the calculation begins. [Note: OKVS is called and all values ​​are set to 0 during encoding], returning P' and r_new. Then r_new (128 bits) and... Give it to PB.

[0109] Step 4.2: PB receives r_new and A', as well as its own B. Calculate... Then calculate . Y'( Send to PA.

[0110] Step 4.3: PA inputs its own data set X, the C it holds, the previously generated r_new, and the received Y', and uses... calculate .

[0111] Step 5: Perform comparison, sorting, and search operations on the deduplicated data within each bucket to obtain the position of the intersection element in the original datasets before deduplication. The comparison, sorting, and search operations are executed within a single bucket, and the operations between buckets can be performed in arbitrary parallelism.

[0112] In one specific implementation, after the OKVS phase, although the intersection elements are within the corresponding buckets, their positions within the buckets are not one-to-one. Therefore, it is necessary to call an equivalence comparison function to compare whether there are identical encrypted values ​​in both encrypted sets, thereby obtaining the position of the intersection in the corresponding set. Note that the corresponding position at this time should be the position of the element after deduplication within each bucket. Since the comparison in this embodiment is based on the robin_hood hash table, the constructed data is out of order. This embodiment uses an efficient sorting method to arrange the data after comparison, and finally calls a lookup function to restore the position information before deduplication.

[0113] It should be noted that in this embodiment, the VOLE stage, OKVS stage, comparison, deduplication, search, and sorting are all performed by a single bucket, and the buckets are computed in parallel without affecting each other.

[0114] Specifically, the intersection positions obtained after comparison are actually out of order because an unordered map is used. Therefore, an efficient sorting algorithm must be used. This embodiment uses the open-source radix sort algorithm.

[0115] The `parallel_radix_sort::SortKeys()` method sorts the positional results. The sorted positions represent the positions in the deduplicated data set; therefore, the index array from the deduplication stage is needed for searching. This embodiment proposes a fast reverse index search method for discrete numerical sets, particularly suitable for scenarios where the target value set is small and the original array is large, such as data filtering, encrypted matching, and privacy queries. First, a mapping relationship is constructed from each value in the input array to all its occurrence positions using the `robin_hood::unordered_map` data structure, ensuring that insertion and search operations have near-constant time complexity. After construction, the method extracts all indices of the target values ​​in the original array according to the specified target values ​​and outputs them in a stable order. Finally, all matching results are linearly stored in a contiguous block of memory and returned to the caller. This embodiment supports features such as duplicate value matching, order preservation, and multi-target merging output, efficiently fulfilling the typical requirements of "multi-key search" while maintaining excellent performance in terms of space and time overhead.

[0116] Example 2:

[0117] Embodiment 2 of the present invention provides a two-party privacy intersection system that supports arbitrary data size, comprising:

[0118] The bucketing module is configured to obtain the original data sets of the two parties that need to interact, determine the initiator and receiver based on the data volume, and bucket the original data of both parties respectively.

[0119] The deduplication module is configured to perform deduplication in parallel for each bucket, retaining the deduplicated data;

[0120] The intersection relationship establishment module is configured to perform security protection operations on a single bucket based on the amount of data possessed by a single bucket after the initiator buckets the data, generating a corresponding transparent vector linear estimate of the number of data.

[0121] The index mapping module is configured to estimate the quantity based on a transparent vector linear estimation, and the initiator and receiver use an unintentional key-value store to perform privacy intersection on the data.

[0122] The adjustment and recovery module is configured to compare, sort, and search the deduplicated data within each bucket to obtain the position of the intersection element in the original datasets of both parties before deduplication.

[0123] Example 3:

[0124] Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed the steps of the two-party privacy intersection method supporting arbitrary data size as described in Embodiment 1 of the present invention.

[0125] Example 4:

[0126] Embodiment 4 of the present invention provides a computer device, the device comprising:

[0127] A processor, adapted to execute computer programs;

[0128] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the two-party privacy intersection method supporting arbitrary data size as described in Embodiment 1 of the present invention.

[0129] The steps and methods involved in Examples 2, 3 and 4 above correspond to those in Example 1. For specific implementation details, please refer to the relevant description section of Example 1.

[0130] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0132] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A two-party privacy intersection method supporting arbitrary data size, characterized in that, Includes the following steps: Obtain the original data sets of the two parties that need to interact, determine the initiator and receiver based on the data volume, and bucket the original data of both parties respectively; Deduplication is performed in parallel for each bucket, and the deduplicated data is retained. Based on the amount of data in a single bucket after the initiator has divided the data into buckets, a corresponding transparent vector linear estimate of the number of data is generated by performing security protection operations on a single bucket. The specific steps for generating a corresponding transparent vector linear estimate of the number of data points in a single bucket based on the amount of data in a single bucket after bucketing by the initiator are as follows: Populate data into buckets whose data volume is less than the threshold. The number of buckets can be set by combining parallel and serial methods; The data within a single bucket is expanded through the expansion phase; The extended data is protected by performing security operations in the VOLE phase, and the number of transparent vector linear estimates of the receiver, i.e., the VOLE B vector, is obtained. Repeatedly perform the expansion and security protection operations to obtain the linear estimate of the initiator's transparent vectors, namely the VOLE A vector and the VOLE C vector; The final generated VOLE A vector and VOLE C vector are written to the MMAP file of PA, and the VOLE B vector is written to the MMAP file of PB. Based on the transparent vector linear estimation quantity, the initiator and receiver use unintentional key-value storage to perform privacy intersection on the data; The deduplicated data within each bucket is compared, sorted, and searched to obtain the position of the intersection element in the original datasets of both sides before deduplication.

2. The two-party privacy intersection method supporting arbitrary data size as described in claim 1, characterized in that, Both sides use the same hash function and modulo operation to bucket their data.

3. The two-party privacy intersection method supporting arbitrary data size as described in claim 2, characterized in that, set up There are n buckets, where n is the specified number of buckets. The first n buckets store the hash result of the original data, and the last n buckets store the original position index of the original data before it was divided into buckets.

4. The two-party privacy intersection method supporting arbitrary data size as described in claim 1, characterized in that, The initiator and receiver are determined based on the amount of data. Specifically, in a balanced scenario, either party can act as the initiator and the other as the receiver. If the difference in the amount of data between the two parties exceeds a set threshold, the party with the smaller amount of data becomes the initiator and the other becomes the receiver.

5. The two-party privacy intersection method supporting arbitrary data size as described in claim 1, characterized in that, Comparison, sorting, and search operations are all executed in a single bucket, and each bucket can be executed in any parallel manner.

6. A two-party privacy intersection system supporting arbitrary data scale, characterized in that, include: The bucketing module is configured to obtain the original data sets of the two parties that need to interact, determine the initiator and receiver based on the data volume, and bucket the original data of both parties respectively. The deduplication module is configured to perform deduplication in parallel for each bucket, retaining the deduplicated data; The intersection relationship establishment module is configured to perform security protection operations on a single bucket based on the amount of data possessed by a single bucket after the initiator buckets the data, generating a corresponding transparent vector linear estimate of the number of data. The specific steps for generating a corresponding transparent vector linear estimate of the number of data points in a single bucket based on the amount of data in a single bucket after bucketing by the initiator are as follows: Populate data into buckets whose data volume is less than the threshold. The number of buckets can be set by combining parallel and serial methods; The data within a single bucket is expanded through the expansion phase; The extended data is protected by performing security operations in the VOLE phase, and the number of transparent vector linear estimates of the receiver, i.e., the VOLE B vector, is obtained. Repeatedly perform the expansion and security protection operations to obtain the linear estimate of the initiator's transparent vectors, namely the VOLE A vector and the VOLE C vector; The final generated VOLE A vector and VOLE C vector are written to the MMAP file of PA, and the VOLE B vector is written to the MMAP file of PB. The index mapping module is configured to estimate the quantity based on a transparent vector linear estimation, and the initiator and receiver use an unintentional key-value store to perform privacy intersection on the data. The adjustment and recovery module is configured to compare, sort, and search the deduplicated data within each bucket to obtain the position of the intersection element in the original datasets of both parties before deduplication.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-5, a two-party privacy intersection method supporting arbitrary data size.

8. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the two-party privacy intersection method supporting arbitrary data size as described in any one of claims 1-5.

Images