Retrieval method and system for privacy protection under large-scale data set

Abstract

The invention relates to a retrieval method and system for privacy protection under a large-scale data set, and belongs to the technical field of information technology and privacy protection. According to the technical scheme, firstly, a pre-screening protocol design method for ensuring user privacy security is provided, so that a system can quickly and accurately position a plurality of most similar data, and the response efficiency of similarity retrieval of privacy protection is greatly improved; and secondly, through a similarity retrieval method for privacy protection, the method can be adapted to any existing two-stage submission protocol, and the existing technology is fully utilized and compatible. Finally, parallel operation is adopted in the design of the protocol, it is guaranteed that the protocol can be equivalently transplanted to a GPU platform for operation, and the computing power of the GPU is exerted to the maximum extent.

Description

Technical Field

[0001] The present invention belongs to the field of information technology and privacy protection technology, and relates to a retrieval method and system for privacy protection under large-scale data sets. Background Art

[0002] In recent years, with the explosive growth of data and the rapid development of information technology (especially network technology and data storage technology), all industries have accumulated massive amounts of data through various channels. Accordingly, people are increasingly concerned about the information security and privacy protection of their own information, requiring that information be protected from leakage while meeting the diversified interactions and full utilization of the current era. Data encryption is one of the important means to protect data privacy. By encrypting data, the security of data during transmission and storage can be ensured. A method that is currently widely adopted is to encrypt the user query and upload it to the server. After receiving the user query, the server decrypts it and then matches the user's query with the entire data set stored on the server one by one to find the most similar data. However, this method exposes the user's query information to the user, threatening the user's privacy, especially when the query information is sensitive information such as the user's biometrics.

[0003] To address the shortcomings of the aforementioned solutions, several cryptographic-based methods have been proposed. Their design aims to allow users to securely query whether an element exists in a specific set while preserving privacy. The core function of private set membership testing is to protect the privacy of both the query content and the data set, ensuring that when users initiate a query, they neither reveal the element they are querying nor obtain any information about the set beyond the query result. However, the efficiency of this process is severely limited by the size of the dataset stored on the server and the performance of the computing platform. First, these solutions all involve complex cryptographic operations and function computations, which incurs very high computational overhead. Second, these solutions require traversing the entire dataset stored on the server, making them unsuitable for large-scale datasets. Finally, these solutions do not consider parallel design and cannot maximize the powerful computing performance of GPUs.

[0004] In current research and practice, optimization primarily focuses on innovation and improvement in cryptographic technology. A common optimization approach is the use of cryptographic tools such as homomorphic encryption and hash functions. These tools allow certain mathematical operations to be performed without decryption, thereby reducing the number of encryption and decryption operations. Furthermore, data structures such as Bloom filters are widely used in implementations. They can accelerate query processing and reduce storage resource consumption by trading space for time. Simultaneously, technologies such as multi-party computation are beginning to be applied to achieve more efficient and secure query operations, further reducing computational complexity. However, these optimization designs also fail to consider the aforementioned traversal costs and design flaws. As a result, when faced with large-scale data, existing solutions still fail to meet the efficiency requirements of practical applications.

[0005] Therefore, it can be seen that existing technical solutions have certain limitations. First, existing solutions often rely on complex cryptographic techniques such as homomorphic encryption, multi-party computation, and searchable encryption. While ensuring privacy and security, these techniques often introduce high computational overhead. Even single queries are very time-consuming, especially when processing large datasets or executing frequent queries. This high computational complexity makes them difficult to directly apply in application scenarios requiring high performance and low latency, such as real-time data querying and ad matching. Second, traditional methods require traversing the entire dataset, which undoubtedly imposes a huge computational burden, especially when dealing with large-scale data. Finally, traditional methods are not easily portable to high-performance GPU computing platforms. Traditional methods are often based on complex cryptographic operations and functions, which are difficult to directly accelerate on GPUs and cannot maximize the performance of high-performance GPU computing platforms. In summary, while existing methods provide effective solutions for privacy protection and security, their high computational complexity, large communication overhead, poor scalability, and low query efficiency limit their widespread application in large-scale practical applications.

[0006] Therefore, there is an urgent need for an efficient retrieval method for large-scale data sets that can significantly improve query efficiency and at the same time well protect user information privacy. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a privacy-preserving retrieval method and system for large-scale data sets. The method and system reduce the overhead of the process and improve the efficiency of the query through a pre-screening protocol, and adopt a privacy-preserving similarity retrieval method to efficiently identify the several data that are most similar to the query and give the similarity between them.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A privacy-preserving retrieval method for a large-scale dataset includes the following steps:

[0010] S1. Design a pre-screening protocol. Given a query and the entire dataset, the protocol uses the statistical characteristics of cosine distance in high-dimensional space to obtain several pieces of data that are as similar as possible to the query from the entire dataset, forming a candidate set. This allows the system to quickly and accurately locate the most similar pieces of data, significantly improving the response efficiency of privacy-preserving similarity retrieval.

[0011] S2. Perform a privacy set membership test. Given a query and the entire dataset, efficiently identify the several pieces of data that are most similar to the query and give the similarity between them.

[0012] S3. According to the similarity situation, retrieve the data information with the highest similarity.

[0013] Furthermore, in step S1, it specifically includes:

[0014] S11. Generate random vectors: Sample from a standard Gaussian distribution to obtain k random vectors. The length of the random vector is the same as the query length m, where k is required to be less than m.

[0015] S12. Let the random vector be R = {r1, r2, ..., rk}, the local data be Y = {y1, y2, ..., yk}, and dij be the cosine distance between the i-th local data and the j-th random vector; calculate the cosine distance between each local data and each random vector generated in S11, denoted as di, which represents the distance between the i-th local data and each random vector;

[0016] S13, calculate the distance between the query x and each random vector, denoted as q, which represents the cosine distance between each random vector in S11 and the query x;

[0017] S14, subtract the distance vector q in S13 from the distance vector di of each local data, and then take the absolute value of the result bit by bit;

[0018] S15. Compare the distance vector obtained in S14 with a preset value bit by bit. If there is a value greater than the preset, the local data corresponding to the distance vector is discarded; otherwise, it is added to the candidate set.

[0019] S16. Output candidate set: The protocol will output a candidate set containing illegal data that is most likely to be similar to the user data.

[0020] Furthermore, in step S2, it specifically includes:

[0021] S21. Given the entire dataset B and the query q, set a threshold and use them as input to the pre-screening protocol defined in step S1. The pre-screening protocol outputs a candidate set S.

[0022] S22. Select any two-phase commit protocol;

[0023] S23. The candidate set S obtained in S21, the query q and the pre-set threshold are used as inputs to the two-phase commit protocol selected in S22.

[0024] S24. Use the output of the protocol in S22 as similarity data.

[0025] Furthermore, in step S1, a pre-screening protocol is further designed with parallel operation to ensure that the protocol can be equivalently ported to the GPU platform for operation, thereby maximizing the computing power of the GPU. Specifically, the pre-screening protocol includes:

[0026] 1) Generate random vectors: Sampling from a standard two-dimensional Gaussian distribution to obtain a random matrix R with a dimension of (m*k);

[0027] 2) Organize the local data into a local data matrix Y by row. The matrix dimension is (n*m). Calculate the multiplication result of matrix R and matrix Y. Then take the value of the arccos function element by element to obtain the distance matrix D.

[0028] 3) Compute the matrix multiplication between the query x and the matrix R, then evaluate the arccos function element-wise, and record the result as q;

[0029] 4) Subtract the preset q from the matrix D row by row and take the absolute value;

[0030] 5) Check the values ​​of the matrix obtained in 4) synchronously by row. If the value of an element in the row is greater than the preset value, the local data corresponding to the row is discarded. Otherwise, the local data corresponding to the row is added to the candidate set.

[0031] The present invention also provides a privacy-protected retrieval system for large-scale data sets.

[0032] The beneficial effects of the present invention are:

[0033] The technical solution of the present invention can bring beneficial effects to the security and efficiency of privacy-preserving similarity retrieval. First, by proposing a pre-screening protocol design method that ensures the privacy security of users, the system can quickly and accurately locate the most similar data, greatly improving the response efficiency of privacy-preserving similarity retrieval. Secondly, this paper proposes a privacy-preserving similarity retrieval method that can adapt to any existing two-phase commit protocol, making full use of and compatible with existing technologies. Finally, the design of the protocol adopts parallel operation, which ensures that the protocol can be equivalently ported to the GPU platform for operation, maximizing the computing power of the GPU.

[0034] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0036] Figure 1 A flow chart of the method for designing the pre-screening protocol of the present invention;

[0037] Figure 2 This is a flow chart of the privacy similarity test design method of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0039] The present invention utilizes a method that combines pre-screening protocol design with privacy set membership testing to significantly improve the efficiency of testing under large-scale data. The present invention provides a privacy-preserving retrieval method under large-scale data sets, which includes the following steps: S1. Designing a pre-screening protocol, given a query and the entire data set, obtaining several pieces of data that are as similar as possible to the query in the entire data set based on the statistical characteristics of the cosine distance in high-dimensional space, and forming a candidate set, so that the system can quickly and accurately locate the most similar pieces of data, significantly improving the response efficiency of privacy-preserving similarity retrieval; S2. Performing a privacy set membership test, given a query and the entire data set, efficiently identifying the pieces of data that are most similar to the query, and giving the similarity between them. S3. Retrieving the data information with the highest similarity based on the similarity situation.

[0040] Existing privacy set membership tests rely on cryptographic operations such as homomorphic encryption, hash functions, and multi-party secure computation, which are usually accompanied by high computational costs. When the data scale expands to millions or hundreds of millions of records, each query requires a large amount of calculation and comparison in the encrypted space, resulting in a significant increase in query response time. Especially in application scenarios that require frequent queries or high real-time requirements, this performance degradation will become more obvious and difficult to meet actual application needs. In step S1 of the present invention, based on the statistical characteristics of cosine distance in high-dimensional space, a pre-screening protocol that can greatly improve test efficiency is proposed, such as Figure 1 As shown, in step S1, it specifically includes:

[0041] S11. Generate random vectors: Sample from a standard Gaussian distribution to obtain k random vectors. The length of the random vector is the same as the query length m, where k is required to be less than m.

[0042] S12. Let the random vector be R = {r1, r2, ..., rk}, the local data be Y = {y1, y2, ..., yk}, and dij be the cosine distance between the i-th local data and the j-th random vector; calculate the cosine distance between each local data and each random vector generated in S11, denoted as di, which represents the distance between the i-th local data and each random vector;

[0043] S13, calculate the distance between the query x and each random vector, denoted as q, which represents the cosine distance between each random vector in S11 and the query x;

[0044] S14, subtract the distance vector q in S13 from the distance vector di of each local data, and then take the absolute value of the result bit by bit;

[0045] S15. Compare the distance vector obtained in S14 with a preset value bit by bit. If there is a value greater than the preset, the local data corresponding to the distance vector is discarded; otherwise, it is added to the candidate set.

[0046] S16. Output candidate set: The protocol will output a candidate set containing illegal data that is most likely to be similar to the user data.

[0047] In addition, in step S1, this protocol does not design complex cryptographic operations. While ensuring equivalence, GPU acceleration can be used to maximize the computing power of the GPU, significantly improve the operating efficiency of the protocol, and ensure that the protocol can be equivalently ported to the GPU platform for operation. Specifically,

[0048] 1) Generate random vectors: Sampling from a standard two-dimensional Gaussian distribution to obtain a random matrix R with a dimension of (m*k);

[0049] 2) Organize the local data into a local data matrix Y by row. The matrix dimension is (n*m). Calculate the multiplication result of matrix R and matrix Y. Then take the value of the arccos function element by element to obtain the distance matrix D.

[0050] 3) Compute the matrix multiplication between the query x and the matrix R, then evaluate the arccos function element-wise, and record the result as q;

[0051] 4) Subtract the preset q from the matrix D row by row and take the absolute value;

[0052] 5) Check the values ​​of the matrix obtained in 4) synchronously by row. If the value of an element in the row is greater than the preset value, the local data corresponding to the row is discarded. Otherwise, the local data corresponding to the row is added to the candidate set.

[0053] In step S2, the traditional privacy similarity test method uses a two-phase commit protocol between each piece of data in the entire data set and the query to securely obtain similar data. This process involves complex encryption and comparison operations, and has high computational complexity. Especially when processing large-scale data, the performance may be poor. In addition, the process usually requires a large amount of encrypted data to be transmitted between the user and the server, which has a large communication overhead and may affect the real-time performance of the system. Finally, the process usually involves some complex cryptographic operations, resulting in limited acceleration effects obtained by using GPUs. Therefore, the present invention provides a design method for privacy similarity testing, such as Figure 2 As shown, the following steps are included:

[0054] S21. Given the entire dataset B and the query q, set a threshold and use them as input to the pre-screening protocol defined in step S1. The pre-screening protocol outputs a candidate set S.

[0055] S22. Select any two-phase commit protocol;

[0056] S23. The candidate set S obtained in S21, the query q and the pre-set threshold are used as inputs to the two-phase commit protocol selected in S22.

[0057] S24. Use the output of the protocol in S22 as similarity data.

[0058] In this embodiment, a specific embodiment of the technical solution of the present invention is as follows:

[0059] Detecting whether user data violates regulations: When users use cloud storage services or cloud computing services, they need to upload their own data to the server for processing. The server needs to confirm whether the data uploaded by the user violates regulations without obtaining the specific content of the user data, while ensuring the efficiency of data processing. In this case, the technical solution of the present invention can be used to solve the problem, which can be divided into two steps, S1 and S2, as follows:

[0060] S1. Use the pre-screening protocol between the server and the user to obtain the candidate set:

[0061] S11. The server generates a random matrix R according to the pre-screening protocol, and calculates the distance matrix D based on the random matrix R and the entire data set B;

[0062] S12. The server sends the random matrix R to the user, and the user uses the random matrix R to calculate the cosine distance q between the query and the random vector;

[0063] S13, q is sent back to the server, and the server selects a candidate set based on q and D;

[0064] S2. Use a two-phase commit protocol between each element in the candidate set and the user data:

[0065] S21, traverse each element in the candidate set and use the selected two-phase commit protocol with the user data respectively;

[0066] S22. Find the most similar pieces of data based on the results in S21;

[0067] S23. If the distance between a certain data in S21 and the user data is less than or equal to the threshold, the user data is in violation; otherwise, the user data is in compliance.

[0068] In summary, the technical solution provided by the present invention enables the system to quickly and accurately locate the most similar data items, significantly improving the response efficiency of privacy-preserving similarity retrieval. It can be adapted to any existing two-phase commit protocol, fully utilizing and being compatible with existing technologies. It also ensures that the protocol can be equivalently ported to a GPU platform for operation, maximizing the computing power of the GPU.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A privacy-preserving retrieval method for large-scale data sets, characterized by: The method comprises the following steps: S1. Design a pre-screening protocol. Given a query and the entire data set, obtain several pieces of data that are as similar as possible to the query in the entire data set based on the statistical characteristics of the cosine distance in high-dimensional space to form a candidate set, so that the system can quickly and accurately locate the most similar pieces of data. S2. Perform a privacy set membership test. Given a query and the entire data set, efficiently identify the several pieces of data that are most similar to the query and give the similarity between them. S3. According to the similarity situation, retrieve the data information with the highest similarity.

2. According to claim 1, a privacy-preserving retrieval method for a large-scale data set is characterized by: In step S1, it specifically includes: S11. Generate random vectors: Sample from a standard Gaussian distribution to obtain k random vectors, the length of which is the same as the length of the query m, where k is required to be less than m; S12, let the random vector be R = {r1, r2, ..., rk}, the local data be Y = {y1, y2, ..., yk}, dij is the cosine distance between the i-th local data and the j-th random vector; calculate the cosine distance between each local data and each random vector generated in S11, denoted as di, which represents the distance between the i-th local data and each random vector; S13, calculating the distance between the query x and each random vector, denoted as q, which represents the cosine distance between each random vector in S11 and the query x; S14, subtract the distance vector q in S13 from the distance vector di of each local data, and then take the absolute value of the result bit by bit; S15, comparing the distance vector obtained in S14 with a preset value bit by bit, if there is a value greater than the preset, the local data corresponding to the distance vector is discarded, otherwise it is added to the candidate set; S16. Output a candidate set: The protocol will output a candidate set containing the illegal data that is most likely to be similar to the user data.

3. The privacy-preserving retrieval method for a large-scale data set according to claim 2, characterized in that: In step S2, it specifically includes: S21, given the entire data set B and the query q, set a threshold, and use them as the input of the pre-screening protocol defined in step S1, and the pre-screening protocol outputs a candidate set S; S22. Select any two-phase commit protocol; S23, taking the candidate set S obtained in S21, the query q and the pre-set threshold as the input of the two-phase commit protocol selected in S22; S24. Use the output of the protocol in S22 as similar data.

4. The privacy-preserving retrieval method for a large-scale dataset according to claim 1, characterized in that: In step S1, a pre-screening protocol is further designed with parallel operation to ensure that the protocol can be equivalently transplanted to the GPU platform for operation, so as to maximize the computing power of the GPU, including: 1) Generate random vectors: Sample from a standard two-dimensional Gaussian distribution to obtain a random matrix R, the dimension of the random matrix is ​​(m*k); 2) The local data is organized into a local data matrix Y by row, the dimension of the matrix is ​​(n*m), the multiplication result of the matrix R and the matrix Y is calculated, and then the value of the arccos function is taken element by element to obtain the distance matrix D; 3) Calculate the matrix multiplication between the query x and the matrix R, then take the value of the arccos function element by element, and record the result as q; 4) Subtract the preset q from the matrix D row by row and take the absolute value; 5) Check the values ​​of the matrix obtained in 4) row by row synchronously. If the value of an element in the row is greater than the preset value, the local data corresponding to the row is discarded, otherwise the local data corresponding to the row is added to the candidate set.

5. A privacy-preserving retrieval system for large-scale data sets, characterized by: The system adopts the method as claimed in any one of claims 1 to 4.

Images