A privacy protection retrieval method and system under a large-scale data set

Abstract

The present application relates to a kind of privacy protection retrieval method and system under large-scale data set, belong to information technology and privacy protection technical field.The present application technical scheme includes: first, by proposing a kind of pre-screening protocol design method for guaranteeing user privacy security, make the system can quickly, accurately locate to most similar data, greatly improve the response efficiency of privacy protection similarity retrieval.Second, by a kind of privacy protection similarity retrieval method, can adapt to any one of existing two-phase commit protocol, make full use of and compatible with existing technology.Finally, the design of the protocol uses parallel operation, ensure that the protocol can be equivalently transplanted to GPU platform and run, maximum degree play out the computing capacity of GPU.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of information technology and privacy protection technology, and relates to a privacy protection retrieval method and system under a large-scale data set. BACKGROUND

[0002] In recent years, with the explosive growth of data, the rapid development of information technology (especially network technology and data storage technology), various industries have accumulated massive data through various channels. Correspondingly, people pay more and more attention to the information security and privacy protection of their own information, and require that information be utilized fully while meeting the diversified interaction of the current era and ensuring that information is not leaked. Data encryption is one of the important means to protect data privacy. Through encryption processing of data, the security of data in the transmission and storage process can be ensured. A widely adopted method is to upload the encrypted user query to the server, and the server decrypts the user query after receiving it, and then matches the user query with the entire data set stored in the server one by one to find the most similar data. However, this method exposes the user's query information to the user, threatening the user's privacy security, especially when the query information is sensitive information such as the user's biological characteristics.

[0003] In order to solve the defects in the above-mentioned scheme, some methods based on cryptography are proposed, and the design purpose is to allow users to safely query whether a certain element exists in a specific set under the premise of protecting privacy. The core function of the privacy set membership test is to realize the privacy protection of the query content and the data set, and to ensure that the user does not leak the element they are querying when initiating the query, nor does the user obtain any information about the set other than the query result. However, the efficiency of this process is severely limited by the size of the data set stored in the server and the performance of the computing platform. First, these schemes all design complex cryptographic operations and function calculations, which brings very high computational overhead. Second, these schemes need to traverse the entire data set stored in the server, which leads to their inability to be applied to large-scale data sets. Finally, these schemes do not consider parallel design and cannot maximize the powerful computing performance of GPU.

[0004] In current research and practice, optimization mainly focuses on the innovation and improvement of cryptographic techniques. A common optimization method is to use cryptographic tools such as homomorphic encryption and hash functions, which allow certain mathematical operations to be performed without decryption, thereby reducing the amount of encryption and decryption operations. In addition, data structures such as Bloom filters are widely used in implementations, which can speed up query processing and reduce storage resource consumption by trading space for time. At the same time, techniques such as multi-party computation are also being applied to achieve more efficient and secure query operations, further reducing computational complexity. However, these optimization designs do not take into account the traversal costs and design flaws mentioned above, which results in existing solutions still failing to meet the efficiency requirements of practical applications when faced with large-scale data.

[0005] Therefore, it can be seen that the existing technical solutions have certain limitations, which are reflected in the following aspects. First, existing solutions usually rely on complex cryptographic techniques such as homomorphic encryption, multi-party computation, searchable encryption, etc. These techniques, while ensuring privacy and security, often introduce high computational overhead. Even for a single query, it is very time-consuming, especially when dealing with large data sets or performing frequent queries, the computational burden is more obvious. This high computational complexity makes it difficult to be directly applied in high-performance and low-latency application scenarios such as real-time data query and advertisement matching. Second, traditional methods require traversal of the entire data set, which undoubtedly brings a huge computational burden, especially when dealing with large-scale data. Finally, traditional methods are not easily migrated to high-performance GPU computing platforms for execution. Traditional methods are usually based on complex cryptographic operations and functions, which are difficult to be directly accelerated by GPU, and cannot maximize the performance of high-performance GPU computing platforms. In summary, although existing methods provide effective solutions in terms of privacy protection and security, their high computational complexity, large communication overhead, poor scalability, and low query efficiency limit their promotion in large-scale practical applications.

[0006] Therefore, there is an urgent need for an efficient retrieval method for large-scale data sets that can significantly improve query efficiency while protecting user information privacy. SUMMARY

[0007] Therefore, the purpose of the present application is to provide a privacy-protected retrieval method and system for large-scale data sets, which reduces the overhead of the process through pre-screening protocols, improves the efficiency of queries, and efficiently identifies the most similar number of data to the query using a privacy-protected similarity retrieval method, and gives the similarity between them.

[0008] To achieve the above purpose, the present application provides the following technical solutions:

[0009] A privacy protection retrieval method under a large-scale data set, the method comprising the following steps:

[0010] S1, a pre-screening protocol is designed, given a query and a whole data set, according to the statistical characteristics of the cosine distance in high-dimensional space, a number of data in the whole data set similar to the query are obtained, and a candidate set is formed, so that the system can quickly and accurately locate the most similar data, and the response efficiency of the privacy protection similarity retrieval is greatly improved;

[0011] S2, performing privacy set member test, given a query and a whole data set, efficiently identifying the most similar data to the query and giving the similarity between them.

[0012] S3, according to the similarity, the data information with the highest similarity is retrieved.

[0013] Further, in step S1, specifically comprising:

[0014] S11, producing a random vector: sampling from a standard Gaussian distribution to obtain k random vectors, the length of the random vector is the same as the length m of the query, wherein k is required to be less than m;

[0015] S12, record the random vector as R={r1,r2,...,rk}, the local data as Y={y1,y2,...,yk}, and dij as the cosine distance between the ith local data and the jth random vector; Calculate the cosine distance between each local data and each random vector generated in S11, denoted as di, indicating the distance between the ith local data and each random vector;

[0016] S13, calculate the distance between the query x and each random vector, denoted as q, indicating the cosine distance between each random vector in S11 and the query x;

[0017] S14, subtract the distance vector q in S13 from the distance vector di of each local data, and take the absolute value of the result bit by bit;

[0018] S15, compare the distance vector obtained in S14 with the preset preset bit by bit, if there is a value greater than the preset, the local data corresponding to the distance vector is discarded, otherwise it is added to the candidate set;

[0019] S16, the candidate set is taken as the output: the protocol will output a candidate set containing the most likely similar illegal data to the user data.

[0020] Further, in step S2, specifically comprising:

[0021] S21, given the whole data set B and the query q, set the threshold value, take them as the input of the pre-screening protocol defined in step S1, and the pre-screening protocol outputs a candidate set S;

[0022] S22, select an arbitrary two-phase commit protocol;

[0023] S23, take the candidate set S obtained in S21, the query q and the pre-set threshold value as the input of the two-phase commit protocol selected in S22;

[0024] S24, the output of the protocol in S22 is similar data.

[0025] Further, in step S1, the pre-screening protocol is further designed to be parallelly operable, so that the protocol can be equivalently transplanted to the GPU platform for running, and the computing capacity of the GPU is maximally exerted, and specifically includes:

[0026] 1) produce a random vector: sample from a standard two-dimensional Gaussian distribution to obtain a random matrix R, and the dimension of the random matrix is (m*k);

[0027] 2) group the local data by rows into a local data matrix Y, the dimension of the matrix is (n*m), calculate the multiplication result of the matrix R and the matrix Y, and then take the value of the arccos function by element to obtain a distance matrix D;

[0028] 3) calculate the matrix multiplication between the query x and the matrix R, and then take the value of the arccos function by element, and record the obtained result as q;

[0029] 4) subtract the preset q from the matrix D by row and take the absolute value;

[0030] 5) synchronously check the value of the matrix obtained in 4) by row, if the value of an element in the row is greater than the preset, the local data corresponding to the row is discarded, otherwise the local data corresponding to the row is added to the candidate set.

[0031] The application also provides a privacy protection retrieval system under a large-scale data set.

[0032] The application has the beneficial effects that:

[0033] The technical scheme of the present application can bring beneficial effects to the security and efficiency of privacy-protected similarity retrieval. First, by proposing a pre-screening protocol design method that guarantees user privacy security, the system can quickly and accurately locate the most similar data, greatly improving the response efficiency of privacy-protected similarity retrieval. Second, the present application proposes a privacy-protected similarity retrieval method that can adapt to any existing two-phase commit protocol, making full use of and compatible with existing technologies. Finally, the design of the protocol adopts parallel operation, ensuring that the protocol can be equivalently transplanted to the GPU platform for operation, maximizing the computing power of the GPU.

[0034] Other advantages, objects, and features of the present application will be understood in view of the following specification, to some extent, will be apparent to those skilled in the art based on the study of the following, or can be taught from the practice of the present application. The objects and other advantages of the present application can be achieved and obtained by the following specification. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred detailed description of the present application will be combined with the drawings as follows, wherein:

[0036] Figure 1 Flow chart of the pre-screening protocol design method of the present application;

[0037] Figure 2 Flow chart of the privacy similarity test design method of the present application. DETAILED DESCRIPTION

[0038] The technical solutions of the present application will be described in detail below in combination with the drawings.

[0039] The present application uses the method of combining pre-screening protocol design and privacy set member test to greatly improve the efficiency of test under large-scale data. The privacy-protected retrieval method provided by the present application under large-scale data set includes the following steps: S1, design a pre-screening protocol, given a query and the whole data set, according to the statistical characteristics of cosine distance in high-dimensional space, get the data in the whole data set that is most similar to the query, form a candidate set, so that the system can quickly and accurately locate the most similar data, greatly improving the response efficiency of privacy-protected similarity retrieval; S2, perform privacy set member test, given a query and the whole data set, efficiently identify the most similar data to the query and give their similarity. S3, according to the similarity, retrieve the data information with the highest similarity.

[0040] Existing privacy set membership test relies on cryptographic operations such as homomorphic encryption, hash functions and multi-party secure computation, which usually accompanies high computational cost. When the data scale expands to millions or hundreds of millions of records, a large number of calculations and comparisons need to be carried out in the encrypted space for each query, resulting in a significant increase in query response time. Especially in the application scenarios that require frequent queries or have high real-time requirements, this performance degradation will become more obvious, which is difficult to meet the actual application requirements. The present application proposes a pre-screening protocol that can greatly improve the test efficiency based on the statistical characteristics of the cosine distance in high-dimensional space in step S1, as shown in the following formula: Figure 1 The step S1 specifically includes the following steps:

[0041] S11, generating a random vector: sampling from a standard Gaussian distribution to obtain k random vectors, the length of the random vector is the same as the length m of the query, and k is required to be less than m;

[0042] S12, the random vector is R={r1, r2,..., rk}, the local data is Y={y1, y2,..., yk}, and dij is the cosine distance between the i-th local data and the j-th random vector; the cosine distance between each local data and each random vector generated in S11 is calculated, denoted as di, which represents the distance between the i-th local data and each random vector;

[0043] S13, calculating the distance between the query x and each random vector, denoted as q, which represents the cosine distance between each random vector in S11 and the query x;

[0044] S14, subtracting the distance vector q in S13 from the distance vector di of each local data, and then taking the absolute value of the obtained result bit by bit;

[0045] S15, comparing the distance vector obtained in S14 with the preset preset bit by bit, if there is a value greater than the preset, the local data corresponding to the distance vector is discarded, otherwise it is added to the candidate set;

[0046] S16, outputting the candidate set: the protocol will output a candidate set containing the most likely illegal data similar to the user data.

[0047] In addition, in step S1, the present protocol does not design complex cryptographic operations, which can be accelerated using GPU under the condition of ensuring equivalence, maximizes the computing power of GPU, greatly improves the running efficiency of the protocol, and ensures that the protocol can be equivalently transplanted to the GPU platform for running, specifically including:

[0048] 1) Produce a random vector: sample from a standard two-dimensional Gaussian distribution to obtain a random matrix R, the dimension of the random matrix is (m*k);

[0049] 2) Group the local data by row to form a local data matrix Y, the dimension of the matrix is (n*m), calculate the multiplication result of matrix R and matrix Y, and then take the arccos function value by element to obtain the distance matrix D;

[0050] 3) Calculate the matrix multiplication between the query x and the matrix R, and then take the arccos function value by element, and record the obtained result as q;

[0051] 4) Subtract the preset q from the matrix D by row and take the absolute value;

[0052] 5) Synchronously check the value of the matrix obtained in 4) by row, if the value of an element in the row is greater than the preset, the local data corresponding to the row is discarded, otherwise the local data corresponding to the row is added to the candidate set.

[0053] In step S2, the traditional privacy similarity test method uses a two-phase commit protocol between each data in the whole data set and the query to securely obtain similar data. This process involves complex encryption and comparison operations, and has high computational complexity. Especially when dealing with large-scale data, the performance may not be good. In addition, this process usually needs to pass a large amount of encrypted data between the user and the server, and the communication overhead is large, which may affect the real-time performance of the system. Finally, this process usually designs some complex cryptographic operations, which limits the acceleration effect obtained by using GPU. Therefore, the present application provides a design method for privacy similarity test, as shown in Figure 2 The method comprises the following steps:

[0054] S21, given the whole data set B and the query q, set the threshold value, and take them as the input of the pre-screening protocol defined in step S1, and the pre-screening protocol outputs a candidate set S;

[0055] S22, select an arbitrary two-phase commit protocol;

[0056] S23, take the candidate set S obtained in S21, the query q and the pre-set threshold value as the input of the two-phase commit protocol selected in S22;

[0057] S24, take the output of the protocol in S22 as the similar data.

[0058] In this embodiment, a specific implementation of the technical solution of the present application is as follows:

[0059] Detecting whether user data is in violation: when a user uses a cloud storage service or a cloud computing service, the user needs to upload his own data to the server side for processing. The server side needs to confirm whether the data uploaded by the user is in violation without obtaining the specific content of the user data, while ensuring the efficiency of data processing. At this time, the technical solution of the present application can be used to solve the problem, which can be divided into S1 and S2 two steps, as follows:

[0060] S1, obtaining a candidate set using a pre-screening protocol between the server and the user:

[0061] S11, the server generates a random matrix R according to the pre-screening protocol, and calculates a distance matrix D according to the random matrix R and the entire data set B;

[0062] S12, the server sends the random matrix R to the user, and the user calculates the cosine distance q between the query and the random vector using the random matrix R;

[0063] S13, send q back to the server, and the server screens out a candidate set according to q and D;

[0064] S2, using a two-phase commit protocol between each element in the candidate set and the user data:

[0065] S21, traverse each element in the candidate set, and use the selected two-phase commit protocol with the user data respectively;

[0066] S22, find a number of data most similar to the user data according to the results in S21;

[0067] S23, if the distance between a certain data in S21 and the user data is less than or equal to a threshold value, the user data is in violation, otherwise the user data is in compliance.

[0068] In summary, the technical solution provided by the present application enables the system to quickly and accurately locate a number of data most similar to the user data, greatly improving the response efficiency of privacy protection similarity search; can adapt to any existing two-phase commit protocol, fully utilizing and compatible with the existing technology; ensures that the protocol can be equivalently transplanted to the GPU platform for operation, and maximizes the computing power of the GPU.

[0069] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified without departing from the purpose and scope of the present application, and all modifications should be included in the scope of the claims of the present application.

Claims

1. A method for privacy-preserving search over large-scale datasets, characterized in that: The method comprises the following steps: S1, designing a pre-screening protocol, giving a query and a whole data set, obtaining a number of data in the whole data set that are as similar as possible to the query according to the statistical characteristics of the cosine distance in a high-dimensional space, and composing a candidate set, so that the system can quickly and accurately locate the most similar data; S2, performing a private set membership test, giving a query and a whole data set, efficiently identifying a number of data that are most similar to the query, and giving the similarity between them; S3, retrieving the data information with the highest similarity according to the similarity condition; In step S1, specifically comprising: S11, producing a random vector: sampling from a standard Gaussian distribution to obtain k random vectors, the length of the random vector being the same as the length m of the query, wherein k is required to be less than m; S12, recording the random vector as R={r1, r2,...,rk}, the local data as Y={y1, y2,...,yk}, and dij as the cosine distance between the ith local data and the jth random vector; calculating the cosine distance between each local data and each random vector produced in S11, denoted as di, indicating the distance between the ith local data and each random vector; S13, calculating the distance between the query x and each random vector, denoted as q, indicating the cosine distance between each random vector in S11 and the query x; S14, subtracting the distance vector q in S13 from the distance vector di of each local data, and then taking the absolute value of the result bit by bit; S15, comparing the distance vector obtained in S14 with the preset preset bit by bit, if there is a value greater than the preset, the local data corresponding to the distance vector is discarded, otherwise it is added to the candidate set; S16, taking the candidate set as the output: the protocol will output a candidate set containing the most likely illegal data similar to the user data; In step S2, specifically comprising: S21, given a whole data set B and a query q, set a threshold, and take them as the input of the pre-screening protocol defined in step S1, and the pre-screening protocol outputs a candidate set S; S22, selecting an arbitrary two-phase commit protocol; S23, taking the candidate set S obtained in S21, the query q and the pre-set threshold as the input of the two-phase commit protocol selected in S22; S24, taking the output of the protocol in S22 as the similar data. 2.The method of claim 1, wherein: In step S1, further adopting a parallel operation to design a pre-screening protocol to ensure that the protocol can be equivalently transplanted to a GPU platform for operation, and maximize the computing power of the GPU, specifically comprising: 1) producing a random vector: sampling from a standard two-dimensional Gaussian distribution to obtain a random matrix R, the dimension of the random matrix being (m*k); 2) grouping the local data into a local data matrix Y by row, the dimension of the matrix being (n*m), calculating the multiplication result of the matrix R and the matrix Y, and then taking the value of the arccos function by element to obtain a distance matrix D; 3) calculating the matrix multiplication between the query x and the matrix R, and then taking the value of the arccos function by element, and recording the obtained result as q; 4) subtract the preset of q from each row of matrix D and take the absolute value; 5) check the value of the matrix obtained in 4) row by row, if the value of an element in a row is greater than the preset, the local data corresponding to the row is discarded, otherwise the local data corresponding to the row is added to the candidate set.

3. A privacy-preserving retrieval system under large-scale dataset, characterized in that: The system employs the method of any one of claims 1 to 2.

Images