A privacy-protected cross-modal semantic retrieval method based on tree structure

Abstract

The application provides a privacy protection cross-modal semantic retrieval method based on a tree structure, and an encrypted index of a tree structure is generated according to image data, wherein the encrypted index of the tree structure can divide image data with a relatively close distance in a high-dimensional space into the same subnode with a very high probability, has high accuracy, and has a very good acceleration effect; meanwhile, since the index and the search token of the application are encrypted, the cross-modal retrieval of the application can allow data retrieval in an encrypted state, thereby protecting the privacy of the data, and even if the data is stored in multiple modes, the data can be protected during transmission and storage, and associated search can be performed between data in different modes.

Description

Technical Field

[0001] This invention belongs to the field of data query and privacy protection technology, and particularly relates to a privacy-preserving cross-modal semantic retrieval method based on a tree structure. Background Technology

[0002] In the contemporary digital age, data sharing has become a dominant trend. Cloud servers offer massive storage space and ultra-fast computing speeds, leading more and more data owners to upload their data to cloud servers to save storage space and facilitate data sharing with other users. However, the complexity of privacy protection continues to escalate in cloud computing and data outsourcing. Traditional encryption techniques make data searching difficult, and achieving search within encrypted text has become a major challenge. To address this challenge, searchable encryption has been proposed, balancing the conflict between search functionality and user privacy. Current searchable encryption methods primarily focus on precise keyword search, fuzzy keyword search, and content-based image retrieval. However, new challenges have emerged.

[0003] With advancements in artificial intelligence, data types have become significantly more diverse. This trend has propelled the integration of image and text systems to the forefront of technological innovation, particularly in the fields of visual and textual information retrieval. For example, on platforms like Unsplash, semantic searches of open-source images can be performed using natural language descriptions. Social media platforms leverage text-based searches to associate relevant images, while e-commerce platforms utilize text-based searches to locate relevant product images. However, searchable encrypted systems that support cross-modal semantic retrieval have yet to be developed.

[0004] The development of image and text retrieval faces three main challenges. First, different modalities exhibit heterogeneity in data representation, data distribution, and cross-modal feature extraction. Second, cross-modal features are high-dimensional vectors. Traditional vector encryption methods, such as homomorphic encryption and secure multi-party computation, are computationally intensive for high-dimensional vectors. Finally, linear search in high-dimensional vector spaces is inefficient. Furthermore, traditional tree index structures such as KD-Tree and ball-tree perform near-linear search operations in high-dimensional vector spaces (a manifestation of the "curse of dimensionality"). Therefore, selecting appropriate feature extraction methods for different modalities and designing effective retrieval methods in encrypted high-dimensional vector spaces remain challenging. Summary of the Invention

[0005] To address the privacy protection issue in cross-modal retrieval, this invention provides a privacy-preserving cross-modal semantic retrieval method based on a tree structure. This method uses an encrypted index within a tree structure to retrieve encrypted images, protecting privacy while achieving cross-modal semantic retrieval with high accuracy and fast search speed.

[0006] A privacy-preserving cross-modal semantic retrieval method based on a tree structure includes the following steps:

[0007] S1: The data owner generates a key and sends the key to the data user through a secure channel;

[0008] S2: The data owner encrypts all image data using a key and sends the encrypted image data, along with an encrypted tree-structured index generated from the image data, to the cloud server for storage.

[0009] S3: Data users generate corresponding search tokens based on the content they are interested in, and send the encrypted search tokens to the cloud server. The cloud server then queries the encrypted index based on the search tokens.

[0010] S4: The cloud server returns the encrypted image data obtained from the query to the data user for decryption, and the data user obtains the image data corresponding to the content of interest.

[0011] Furthermore, the key is:

[0012] sk = (ks, M1, M2)

[0013] Where sk is the key, ks is the symmetric encryption key, M1 and M2 are two different invertible matrices with dimensions of (m+4)×(m+4), and m is the dimension of the image data to be encrypted.

[0014] Furthermore, the method by which the data owner generates a tree-structured encrypted index based on the image data is as follows:

[0015] S21: Define the structure vector of each tree node in the tree structure as node = (ln, rn, lt, rt, data), where the variable ln stores a pointer to the left node, the variable rn stores a pointer to the right node, the variable lt stores judgment data related to the left node, the variable rt stores judgment data related to the right node, the variable data in the leaf node stores the encrypted image vector, and the variable data in non-leaf nodes is empty; the root node is the first level of the tree structure, and the leaf nodes are the last level of the tree structure;

[0016] S22: Determine whether the number of elements in the current image vector set S that the current node needs to process is less than the set threshold N0. If yes, proceed to step S23; if no, proceed to step S24. The initial value of the current node is the root node.

[0017] S23: Encrypt each element p in the current image vector set S. i The resulting encrypted image vector P iStore the data in the variable data of the leaf node of the current node to complete the index construction from the root node to the current leaf node;

[0018] S24: Obtain the child node to which each element in the current image vector set S belongs relative to the current node, and complete the assignment of the structure vector node of the current node, specifically:

[0019] S241: Determine if the current node's level in the tree structure is less than 3. If yes, use the 2-Means algorithm to process the elements p in the current image vector set S. i After clustering, two class vectors p are obtained. a and p b If not, directly select two category vectors p randomly. a and p b ;

[0020] S242: Obtain two class vectors p a and p b The center vector p c =(p a +p b ) / 2, then obtain the element p in the current image vector set S. i The corresponding category distance dis i =(p i -p c )·(p a -p c ), will dis i The element p > 0 i It belongs to the left child node of the current node, and dis i The element p > 0 i Divide into the left subtree set S1, and then divide dis i Element p ≤ 0 i It belongs to the right child node of the current node, and dis i Element p ≤ 0 i Divide it into the right subtree set S2;

[0021] S243: Encrypt two class vectors p a and p b This corresponds to the encryption category vector P. a and encryption category vector P b ;

[0022] S244: Set the current node's variable ln to its left child node, the current node's variable rn to its right child node, and the current node's variable lt to store the encrypted category vector P. a The variable rt of the current node stores the encrypted category vector P. b ;

[0023] S25: Increment the current node level by 1, and re-execute steps S22 to S27, using the left subtree set S1 and the right subtree set S2 as the new current image vector set S respectively, until the encrypted image vectors corresponding to all elements in the original image vector set are stored in the variable data of the corresponding leaf node, thus completing the construction of the encrypted index of the tree structure; where, when the left subtree set S1 is used as the new current image vector set S, the new current node is the left child node of the original current node, and when the right subtree set S2 is used as the new current image vector set S, the new current node is the right child node of the original current node.

[0024] Furthermore, in step S23, each element p in the current image vector set S is encrypted. i The method is as follows:

[0025] For each element p in the current image vector set S i Perform the first encryption operation to obtain the encrypted image vector P corresponding to each element. i The first encryption operation is as follows:

[0026] element p i Expanded to (m+4) dimensional vector Where r1 is a random number, m is the dimension of the image data to be encrypted; the data owner randomly generates a vector p′ with the main diagonal elements as its vector. i And a lower triangular random matrix U with dimensions of (m+4)×(m+4) i One of the main diagonal elements is a vector (1,1,…,1). 1×(m+4) And a lower triangular matrix I with dimensions (m+4)×(m+4) x Using random matrices M1 and M2, each with dimension (m+4)×(m+4), U i Encrypt the image vector P i =M1×I x ×U i ×I x ×M2.

[0027] Furthermore, the category vector p in step S243 a and p b The encryption method is as follows:

[0028] p a and p b Expanded into (m+4) dimensional vectors respectively and Where r1 is a random number; the data owner randomly generates a vector p′ with diagonal elements as its main diagonal. a And a lower triangular random matrix U with dimensions of (m+4)×(m+4) a, a lower triangular matrix I whose main diagonal elements are the vector (1,1,…,1) 1×(m+4) and whose dimension is (m+4)×(m+4) a , a lower triangular random matrix U whose main diagonal elements are the vector p′ b and whose dimension is (m+4)×(m+4) b , a lower triangular matrix I whose main diagonal elements are the vector (1,1,…,1) 1×(m+4) and whose dimension is (m+4)×(m+4) b ; the random matrices M1 and M2 both with the dimension of (m+4)×(m+4) are used to respectively encrypt U a and U b into encrypted class vectors P a =M1×I a ×U a ×I a ×M2 and P b =M1×I b ×U b ×I b ×M2.

[0029] Further, in step S3, the method for the cloud server to query the encrypted index based on the search token is as follows:

[0030] S31: Determine whether the first node in the current priority queue is a leaf node. If yes, add the encrypted image vector P stored in the variable data of the current leaf node i to the result set R; if no, perform step S32; wherein the initial value of the first node is the root node;

[0031] S32: Take out the encrypted class vector P stored in the variable lt of the current first node a and the encrypted class vector P stored in the variable rt b , and calculate the left trace dL=tr(p a ×Q) and the right trace dR=tr(P b ×Q), wherein the tr(*) function represents calculating the trace of the matrix *, and Q is the encrypted search token;

[0032] S33: Update the priority queue according to the magnitudes of dL and dR, specifically:

[0033] Determine whether the absolute value of the difference between dL and dR is less than the set threshold w. If yes, add the left child node and the right child node of the current first node to the priority queue; if no, compare the magnitudes of dL and dR. If dL>dR, add the left child node of the current first node to the priority queue; if dL<dR, add the right child node of the current first node to the priority queue;

[0034] S34: Use the updated priority queue as the new current priority queue and re-execute steps S31 to S33 until the priority queue is empty, then proceed to step S35;

[0035] S35: Calculate the encrypted image vector P in the result set R respectively. i The traces of the product of the search token Q and the top k largest traces will correspond to the encrypted image vector P. i As the final query result.

[0036] Furthermore, the encrypted image vector P is obtained under encrypted conditions. i The method for calculating the Euclidean distance to the search token Q is as follows:

[0037] The CLIP deep neural network model is used to analyze the image data. i Encoded as an m-dimensional vector p i , where i∈[1,n], and n is the number of image data; the CLIP deep neural network model is used to generate corresponding search tokens for the content of interest, where the search token is an m-dimensional vector q;

[0038] Let the m-dimensional vector p i If q directly executes steps S2 to S4, then the encrypted image vector P in step S45... i The trace of the product of the search token Q and the search token Q is the encrypted image vector P. i The Euclidean distance between the search token Q and the search token Q.

[0039] Furthermore, the encrypted image vector P is obtained under encrypted conditions. i The method for cosine similarity with search token Q is as follows:

[0040] The CLIP deep neural network model is used to analyze the image data. i Encoded as an m-dimensional vector p i , where i∈[1,n], and n is the number of image data; the CLIP deep neural network model is used to generate corresponding search tokens for the content of interest, where the search token is an m-dimensional vector q;

[0041] Let the m-dimensional vector p i Perform normalization operation Normalization operation on vector q Then, steps S2 to S4 are executed, and the encrypted image vector P in step S45 is... i The trace of the product of the search token Q and the search token Q is the encrypted image vector P. i Cosine similarity with search token Q.

[0042] Furthermore, the method for generating the search token is as follows:

[0043] Data users generate corresponding search tokens for content of interest using the CLIP deep neural network model, where the search token is an m-dimensional vector q;

[0044] The encryption method for the search token is as follows:

[0045] Expand the search token q into an (m+4) dimensional vector. Where r2 is a random number, m is the dimension of the image data to be encrypted; the data user randomly generates a lower triangular random matrix Y with a main diagonal element of vector q′ and a dimension of (m+4)×(m+4), and a main diagonal element of vector (1,1,…,1). 1×(m+4) And a lower triangular matrix I with dimensions (m+4)×(m+4) y The search token q is encrypted using random matrices M1 and M2, each with dimension (m+4)×(m+4).

[0046] Beneficial effects:

[0047] 1. This invention provides a privacy-preserving cross-modal semantic retrieval method based on a tree structure. It generates an encrypted tree-structured index based on image data. This encrypted index can group closely spaced image data in a high-dimensional space into the same child node with extremely high probability, achieving high accuracy and significant acceleration. Furthermore, since both the index and search token are encrypted, the cross-modal retrieval method allows data retrieval under encrypted conditions, thus protecting data privacy. Even if data is stored across multiple modalities, it ensures data protection during transmission and storage, and enables correlated searches between data from different modalities.

[0048] 2. This invention provides a privacy-preserving cross-modal semantic retrieval method based on a tree structure, which supports both text-based retrieval and content-based image search, and can provide richer and more accurate retrieval results.

[0049] 3. This invention provides a privacy-preserving cross-modal semantic retrieval method based on a tree structure, which can obtain both Euclidean distance and cosine similarity as similarity measures under encrypted conditions, and has strong practicality in real-world scenarios.

[0050] 4. This invention provides a privacy-preserving cross-modal semantic retrieval method based on a tree structure. By using the CLIP deep neural network model, image data of different modalities are mapped to a unified vector space, thereby enabling image data of different modalities to be compared and searched in the same space. Attached Figure Description

[0051] Figure 1A diagram illustrating a privacy-preserving cross-modal semantic retrieval model based on a tree structure, provided by this invention;

[0052] Figure 2 This is a schematic diagram of image and text retrieval for a privacy-preserving cross-modal semantic retrieval method based on a tree structure provided by the present invention;

[0053] Figure 3 A schematic diagram illustrating the binary tree generation for a privacy-preserving cross-modal semantic retrieval method based on a tree structure provided by this invention;

[0054] Figure 4 The simulation results of the privacy-preserving cross-modal semantic retrieval method based on tree structure provided by this invention are shown in four datasets: Imagenet, MNIST, MSCOCO, and fkr30k. Detailed Implementation

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0056] Most existing searchable encryption technologies support keywords but do not support more advanced query services. However, with the development of data diversity, more and more scenarios require cross-modal semantic retrieval, such as image and text retrieval. Because different modalities are heterogeneous, cross-modal retrieval is very challenging. At the same time, protecting data privacy during the query process is also very difficult.

[0057] Therefore, to address the privacy challenges of existing cross-modal semantic retrieval methods, this invention proposes a privacy-preserving cross-modal semantic retrieval method based on a tree structure, which includes a data owner, a cloud server, and a data user. The data owner possesses a large amount of image data, and local storage space limits the use and storage of this image data. Therefore, the data owner chooses to upload the image data to the cloud server for storage, and usually encrypts the images before uploading. The data user can request access to the data from the data owner. The data user can generate a search token based on text or images and search for image data by sending a search request to the cloud server.

[0058] like Figure 1 As shown, a privacy-preserving cross-modal semantic retrieval method based on a tree structure mainly includes initialization, index building, and data querying, specifically including the following steps:

[0059] S1: The data owner generates a key and sends the key to the data user through a secure channel;

[0060] The key is:

[0061] sk = (ks, M1, M2)

[0062] Where sk is the key, ks is the symmetric encryption key, M1 and M2 are two different invertible matrices with dimensions of (m+4)×(m+4), and m is the dimension of the image data to be encrypted.

[0063] S2: The data owner encrypts all image data using a key and sends the encrypted image data, along with an encrypted tree-structured index generated from the image data, to the cloud server for storage.

[0064] Given a set of images I = {I i All image data C, i∈[1,n]}, are encrypted using a symmetric encryption algorithm. i =Sym.Enc(ks,I i ), i∈[1,n]; where n is the number of images; images {C i}, i∈[1,n] are sent to the cloud server for storage.

[0065] The method by which data owners generate a tree-structured encrypted index based on image data specifically includes the following steps:

[0066] S21: Define the structure vector of each tree node in the tree structure as node = (ln, rn, lt, rt, data), where the variable ln stores a pointer to the left node, the variable rn stores a pointer to the right node, the variable lt stores judgment data related to the left node, the variable rt stores judgment data related to the right node, the variable data in the leaf node stores the encrypted image vector, and the variable data in non-leaf nodes is empty; the root node is the first level of the tree structure, and the leaf nodes are the last level of the tree structure;

[0067] S22: Determine whether the number of elements in the current image vector set S that the current node needs to process is less than the set threshold N0. If yes, proceed to step S23; if no, proceed to step S24. The initial value of the current node is the root node.

[0068] S23: Encrypt each element p in the current image vector set S. i The resulting encrypted image vector P i Store the data in the variable data of the leaf node of the current node to complete the index construction from the root node to the current leaf node;

[0069] In this context, each element p in the current image vector set S is encrypted. i The method is as follows:

[0070] For each element p in the current image vector set S iPerform the first encryption operation to obtain the encrypted image vector P corresponding to each element. i The first encryption operation is as follows:

[0071] element p i Expanded to (m+4) dimensional vector Where r1 is a random number, m is the dimension of the image data to be encrypted; the data owner randomly generates a vector p′ with the main diagonal elements as its vector. i And a lower triangular random matrix U with dimensions of (m+4)×(m+4) i One of the main diagonal elements is a vector (1,1,…,1). 1×(m+4) And a lower triangular matrix I with dimensions (m+4)×(m+4) x Using random matrices M1 and M2, each with dimension (m+4)×(m+4), U i Encrypt the image vector P i =M1×I x ×U i ×I x ×M2.

[0072] S24: Obtain the child node to which each element in the current image vector set S belongs relative to the current node, and complete the assignment of the structure vector node of the current node, specifically:

[0073] S241: Determine if the current node's level in the tree structure is less than 3. If yes, use the 2-Means algorithm (a simplification of the k-Means algorithm that divides the data into two classes) to process the elements p in the current image vector set S. i After clustering, two class vectors p are obtained. a and p b If not, directly select two category vectors p randomly. a and p b ;

[0074] S242: Obtain two class vectors p a and p b The center vector p c =(p a +p b ) / 2, then obtain the element p in the current image vector set S. i The corresponding category distance dis i =(p i -p c )·(p a -p c ), will dis i The element p > 0 i It belongs to the left child node of the current node, and disi The element p > 0 i Divide into the left subtree set S1, and then divide dis i Element p ≤ 0 i It belongs to the right child node of the current node, and dis i Element p ≤ 0 i Divide it into the right subtree set S2;

[0075] S243: Encrypt two class vectors p a and p b This corresponds to the encryption category vector P. a and encryption category vector P b The details are as follows:

[0076] p b and p b Expanded into (m+4) dimensional vectors respectively and Where r1 is a random number; the data owner randomly generates a vector p′ with diagonal elements as its main diagonal. a And a lower triangular random matrix U with dimensions of (m+4)×(m+4) a One of the main diagonal elements is a vector (1,1,…,1). 1×(m+4) And a lower triangular matrix I with dimensions (m+4)×(m+4) a One of the main diagonal elements is a vector p′ b And a lower triangular random matrix U with dimensions of (m+4)×(m+4) b One of the main diagonal elements is a vector (1,1,…,1). 1×(m+4) And a lower triangular matrix I with dimensions (m+4)×(m+4) b Using random matrices M1 and M2, each with dimension (m+4)×(m+4), U a and U b Encrypt them respectively into encryption category vector P a =M1×I a ×U a ×I a ×M2 and I b =M1×I b ×U b ×I b ×M2.

[0077] S244: Set the current node's variable ln to its left child node, the current node's variable rn to its right child node, and the current node's variable lt to store the encrypted category vector P. a The variable rt of the current node stores the encrypted category vector P. b ;

[0078] S25: Increment the current node level by 1, and re-execute steps S22 to S27, using the left subtree set S1 and the right subtree set S2 as the new current image vector set S respectively, until the encrypted image vectors corresponding to all elements in the original image vector set are stored in the variable data of the corresponding leaf node, thus completing the construction of the encrypted index of the tree structure; where, when the left subtree set S1 is used as the new current image vector set S, the new current node is the left child node of the original current node, and when the right subtree set S2 is used as the new current image vector set S, the new current node is the right child node of the original current node.

[0079] S3: Data users generate corresponding search tokens based on the content they are interested in, and send the encrypted search tokens to the cloud server. The cloud server then queries the encrypted index based on the search tokens.

[0080] It should be noted that data users can choose either content-based image retrieval or image-text retrieval. After generating a search token based on the desired query content, the token is sent to the cloud server. The cloud server then queries the encrypted forest and returns the results to the data user. Therefore, the method for generating the search token is as follows:

[0081] Data users generate corresponding search tokens for content of interest using the CLIP deep neural network model, where the search token is an m-dimensional vector q;

[0082] The encryption method for the search token is as follows:

[0083] Expand the search token q into an (m+4) dimensional vector. Where r2 is a random number, m is the dimension of the image data to be encrypted; the data user randomly generates a lower triangular random matrix Y with a main diagonal element of vector q′ and a dimension of (m+4)×(m+4), and a main diagonal element of vector (1,1,…,1). 1×(m+4) And a lower triangular matrix I with dimensions (m+4)×(m+4) y The search token q is encrypted using random matrices M1 and M2, each with dimension (m+4)×(m+4).

[0084] The method for cloud servers to query encrypted indexes based on search tokens includes the following steps:

[0085] S31: Determine if the first node in the current priority queue is a leaf node. If it is, store the encrypted image vector P in the variable data of the current leaf node. i Add to result set R; if not, proceed to step S32; where the initial value of the first node is the root node;

[0086] S32: Extract the encrypted category vector P stored in the variable lt of the current first node a and the encrypted category vector P stored in the variable rt b , and calculate the left trace dL=tr(P a ×Q) and the right trace dR=tr(P b ×Q, where the tr(*) function represents calculating the trace of matrix *, that is, the sum of the diagonal elements of the matrix, and Q is the encrypted search token;

[0087] S33: Update the priority queue according to the values of dL and dR, specifically as follows:

[0088] Determine whether the absolute value of the difference between dL and dR is less than a set threshold w, that is, |dL-dR|<w. If yes, the current first node is very close to both the left and right center points, add the left child node and the right child node of the current first node to the priority queue; if no, compare the values of dL and dR, if dL>dR, add the left child node of the current first node to the priority queue, if dL<dR, add the right child node of the current first node to the priority queue;

[0089] S34: Use the updated priority queue as the new current priority queue to re-execute steps S31 to S33 until the priority queue is empty, then proceed to step S35;

[0090] S35: Respectively calculate the trace of the product of the encrypted image vector P in the result set R i and the search token Q, and take the top k encrypted image vectors P corresponding to the largest traces i as the final query result.

[0091] S4: The cloud server returns the queried encrypted image data to the data user for decryption, and the data user obtains the image data corresponding to the content of interest.

[0092] It should be noted that the present invention can also directly obtain the Euclidean distance and cosine similarity between the image vector and the search token when they are in the ciphertext state.

[0093] Specifically, the method for obtaining the Euclidean distance between the encrypted image vector P in the encrypted state i and the search token Q is as follows:

[0094] Use the CLIP deep neural network model to encode each piece of image data I i into an m-dimensional vector p i , where i∈[1,n], n is the number of image data; use the CLIP deep neural network model to generate the corresponding search token from the content of interest, where the search token is an m-dimensional vector q;

[0095] Let the m-dimensional vector p i If q directly executes steps S2 to S4, then the encrypted image vector P in step S45... i The trace of the product of the search token Q and the search token Q is the encrypted image vector P. i The Euclidean distance between the search token Q and the search token Q is:

[0096]

[0097] -0.5(pq) 2 This is the method for calculating the Euclidean distance between the image vector and the search token when they are in plaintext.

[0098] Obtain the encrypted image vector P under encrypted conditions. i The method for cosine similarity with search token Q is as follows:

[0099] The CLIP deep neural network model is used to analyze the image data. i Encoded as an m-dimensional vector p i , where i∈[1,n], and n is the number of image data; the CLIP deep neural network model is used to generate corresponding search tokens for the content of interest, where the search token is an m-dimensional vector q;

[0100] Let the m-dimensional vector p i Perform normalization operation Normalization operation on vector q Then, steps S2 to S4 are executed, and the encrypted image vector P in step S45 is... i The trace of the product of the search token Q and the search token Q is the encrypted image vector P. i The cosine similarity between the search token Q and the search token Q is:

[0101] tr(P×Q)=[p / ||p||2,-0.5,1]·[q / ||q||2,1,-0.5]

[0102] =(p·q) / (||p||2||q||2)-1

[0103] =cos(p,q)-1

[0104] cos(p,q)-1 is the method for calculating the cosine similarity between the image vector and the search token when they are in plaintext.

[0105] It can be seen that the larger the trace, the smaller the Euclidean distance or the greater the cosine similarity.

[0106] In other words, if the m-dimensional vector p i If steps S2 to S4 are executed directly, then the encrypted image vector P in step S45 will be... iThe trace of the product of the search token Q and the search token Q is the encrypted image vector P. i The Euclidean distance between the search token Q and the m-dimensional vector p; i Perform normalization operation Then, steps S2 to S4 are executed, and the encrypted image vector P in step S45 is... i The trace of the product of the search token Q and the search token Q is the encrypted image vector P. i Cosine similarity with search token Q.

[0107] Thus, a privacy-preserving cross-modal semantic retrieval method based on a tree structure has been completed.

[0108] The following section provides a further explanation of the privacy-preserving cross-modal semantic retrieval method based on a tree structure provided by this invention, using specific application scenarios as examples.

[0109] System model diagram as follows Figure 1 Data owners possess a large amount of image data, and local storage space limits the use and storage of this data. Therefore, data owners choose to upload the image data to a cloud server for storage, usually encrypting the images before uploading. Data users can request access permissions from the data owner. Data users can generate search tokens based on text or images and search for image data by sending search requests to the cloud server.

[0110] Suppose a user wants to perform an image-text search operation, and uses cosine similarity as the similarity measurement function.

[0111] Assume the data owner has 8 images (in reality, there could be tens of thousands or hundreds of millions of images); assume the vector dimension is 2;

[0112] Assume the data user wants the most similar data, i.e., top-k = 1;

[0113] Step 1. Initialize the system and generate a key. The data owner generates a key based on the system security parameters, and the key is sent to the data owner via a secure channel. This includes the following sub-steps:

[0114] Step 1.1: Initialize the system and define the structure of the tree node as node = (ln, rn, lt, rt, data);

[0115] Here, ln is a pointer to the left node, rn is a pointer to the right node, lt is the judgment data related to the left node, rt is the judgment data related to the right node, and data is the storage information of the leaf node; this item is empty for non-leaf nodes.

[0116] Step 1.2: Define system parameters N0 = 1, threshold w = 0.2, and similarity measurement function df = cosine similarity;

[0117] Step 1.2: The data owner randomly generates a key sk;

[0118] Where sk = (ks, M1, M2), ks is a symmetric encryption key used to encrypt image data, and M1 and M2 are two 6×6 invertible matrices used for encryption.

[0119] Step 1.3: Send the key sk to the data user through a secure channel;

[0120] Step 2. Build a secure index; specifically: the data owner encrypts all image data and generates an encrypted index based on the image data, including the following sub-steps:

[0121] Step 2.1: Given an image set I = {I1, i ∈ [1, 8]}, such as Figure 2 As shown, there are 8 images; all image data C is encrypted using a symmetric encryption algorithm. i =Sym.Enc(ks,I i ), i∈[1,8];

[0122] Step 2.2: For each image data I i Using the CLIP deep neural network model to I i Encoded as an m-dimensional vector p i Normalize the vector. Put all vectors p i The set is S =

[0123] [0.63826994, 0.76981263], [0.98202917, 0.18872918],

[0124] [0.93591069 0.3522374], [0.46777777, 0.88695246]

[0125] [0.34609858, 0.93819815], [0.99127382, 0.13181886]

[0126] [0.99976908, 0.02148911], [0.55742012, 0.83023057]

[0127] like Figure 3 The diagram shows 8 data points;

[0128] Step 2.3: Instantiate a new node as root; recursively construct a cryptographic tree according to the following steps, passing in the node root, the image vector set S, and the level number level=1:

[0129] Step 2.3.1: Determine if the number of elements in the current image vector set S is less than parameter N0 = 1. If the number of elements in the image vector set S is greater than parameter N0, proceed to step 2.3.2. If it is less than N0, perform the following operation: for each element p in the image vector set S... i Expanded to (m+4) dimensional vector Where r1 is a random number; the data owner generates a random 6×6 vector p′ with the main diagonal elements as its base. i The lower triangular random matrix U i A random 6×6 lower triangular matrix I whose main diagonal elements are vectors (1,1,…,1). x Finally, U is encrypted using random matrices M1 and M2. i For P i =M1×I x ×U i ×I x ×M2; Store all encrypted vectors in the data variable of the leaf node; Construction of this path is complete;

[0130] Step 2.3.2: If the current layer number is less than 3, use the 2-Means algorithm to cluster the image vector dataset S and generate two center points p. a =[0.51392955,0.84608045] and p b =[0.95535211,0.2382454]; for example Figure 3 The marked p a and p b ;

[0131] Step 2.3.3: Generate a hyperplane based on the two selected data points to divide the data into two parts, and calculate the center point of the two data points. like Figure 3 As shown; all data points p in the image vector dataset S i Calculate dis i =(p i -p c )·(p a -p c ); if dis i >0, p i If it is assigned to S1, then dis i <0, p i They are assigned to S2; S1 contains {p1,p4,p5,p8}, and S2 contains {p2,p3,p6,p7}.

[0132] Step 2.3.4: Encrypt the two central nodes. a Expanded into a 6-dimensional vector p′ a =[pa [,-0.5,1,0.2,-0.2]; The data owner generates a random 6×6 vector p′ with the main diagonal elements as its vector. a The lower triangular random matrix U a A random 6×6 lower triangular matrix I whose main diagonal elements are vectors (1,1,…,1). a Finally, U is encrypted using random matrices M1 and M2. a For P a =M1×I a ×U a ×I a ×M2;p b Perform the same operation to encrypt P b ;

[0133] Step 2.3.5: Instantiate two nodes: left node and right node; set the root pointer ln to left node, set the root pointer rn to right node, and store the matrix P in the root variable lt. a The root variable rt stores the matrix P. b ;

[0134] Step 2.3.6: Generate the left subtree. Repeat steps 2.3.1-2.3.5, passing in the vector, node leftnode, image vector set S1, and level+1.

[0135] Step 2.3.7: Generate the right subtree. Repeat steps 2.3.1-2.3.5, passing in the vector, node rightnode, image vector set S2, and level+1.

[0136] Step 2.4: Repeat step 2.3 to generate one subtree, such as... Figure 3 The three-layered encrypted tree shown constitutes a encrypted forest; and the encrypted forest and the encrypted image {C} are then combined. i}, i∈[1,6], send to the cloud server for storage;

[0137] Step 3: Query data; the data user selects image and text search; after generating a search token based on the content to be queried, it is sent to the cloud server, the cloud server queries the encrypted forest and returns the results to the data user.

[0138] Step 3.1: As Figure 2As shown, data users encode the text they want to query, text = 'Aman on a skateboard with a dog outside', into an m-dimensional vector q = [0.3, 0.4] using the CLIP model. If the similarity metric is Euclidean distance, the vector remains unchanged; if the similarity metric is cosine similarity, the vector is normalized.

[0139] Step 3.2: Encrypt the query vector, generate a search token, and send it to the cloud server for searching. The encryption steps are as follows: q is expanded into a 6-dimensional vector q′=[0.6,0.8,1,-0.5,0.3,0.3]. The data user generates a random 6×6 lower triangular matrix Y with the main diagonal elements of vector q′, and a random 6×6 lower triangular matrix I with the main diagonal elements of vector (1,1,…,1). y Finally, Q is encrypted using random matrices M1 and M2. The encrypted matrix Q is sent to the cloud server for querying;

[0140] Step 3.3: The cloud server uses the search token to perform a query, adds the root node of the tree to the priority queue, and repeats the following steps until the priority queue is empty;

[0141] Step 3.3.1: Retrieve the first node from the queue; if node is a leaf node, add node's data to the result set R;

[0142] Step 3.3.2: Retrieve the lt storage content M of node. l and node's rt storage content M r ; Calculate dL = tr(M) l ×Q)=-0.01477791 and dR=tr(M r ×Q)=-0.236192414;

[0143] Step 3.3.3: |dL-dR|>w, compare the size of dL and dR. If dL>dR, add the left node to the priority queue.

[0144] Step 3.3.4: Repeat steps 3.3.1-3.3.3 until the priority queue is empty;

[0145] Step 3.4: Calculate matrices P1 and P8 in R, calculate the traces of their products with the search token Q to obtain -0.00118 and -0.00136 respectively, sort them to obtain the maximum result, and send the corresponding encrypted image 1 to the data user.

[0146] Step 3.5: The data user decrypts the received image to obtain the final result;

[0147] Thus, this invention achieves privacy-preserving cross-modal semantic retrieval, enabling the search for corresponding images through text. Furthermore, this invention supports two retrieval tasks: text-based image retrieval and content-based image search. It also supports two similarity metrics, making it highly practical in real-world scenarios. Figure 3 This paper compares the present invention with Ball trees on four different datasets. As can be seen from the figures, Ball trees do not support high speedup ratios, and at the same speedup ratio, their accuracy is poor. In other words, traditional tree-structured indexes, such as KD trees and Ball trees, suffer from the curse of dimensionality in high-dimensional vector spaces, meaning their search performance is the same as linear search and they do not provide a significant speedup effect. The tree structure of the present invention can group closely spaced image data into the same child node with a very high probability in high-dimensional space, achieving high accuracy and a significant speedup effect.

[0148] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A privacy-preserving cross-modal semantic retrieval method based on a tree structure, characterized in that, Includes the following steps: S1: The data owner generates a key and sends the key to the data user through a secure channel; S2: The data owner encrypts all image data using a key and sends the encrypted image data, along with an encrypted tree-structured index generated from the image data, to the cloud server for storage. S3: Data users generate corresponding search tokens based on the content they are interested in, and send the encrypted search tokens to the cloud server. The cloud server then queries the encrypted index based on the search tokens. S4: The cloud server returns the encrypted image data obtained from the query to the data user for decryption, and the data user obtains the image data corresponding to the content of interest; The data owner generates a tree-structured encrypted index based on the image data using the following method: S21: Define the structure vector of each node in the tree structure. , where variables Used to store a pointer to the left node, variable Used to store a pointer to the right node, variable Used to store judgment data related to the left node, variables Used to store decision data related to the right node; variables in the leaf nodes. Variables used to store the encrypted image vector, not leaf nodes. Empty; the root node is the first level of the tree structure, and the leaf nodes are the last level of the tree structure; S22: Determine the current image vector set that needs to be processed at the current node. Is the number of elements less than a set threshold? If yes, proceed to step S23; if no, proceed to step S24; where the initial value of the current node is the root node. S23: Encrypt the current image vector set Each element in The resulting encrypted image vector Variables stored in the leaf node of the current node In the middle, complete the index construction from the root node to the current leaf node; S24: Obtain the current image vector set Each element in the vector is relative to the child node to which the current node belongs, thus completing the structure vector of the current node. The assignment is as follows: S241: Determine if the current node's level in the tree structure is less than 3. If yes, use the 2-Means algorithm to process the current image vector set. elements in After clustering, two category vectors are obtained. and If not, directly select two category vectors randomly. and ; S242: Obtain two class vectors and center vector Then obtain the current image vector set. elements in Corresponding category distance ,Will elements Belongs to the left child of the current node, and will elements Partition into the left subtree set ,Will elements Belongs to the right child node of the current node, and will elements Partition into the right subtree set ; S243: Encrypt two class vectors and This corresponds to the encryption category vector. and encryption category vector ; S244: Set the variable of the current node. A pointer to the left child node of the current node. A pointer to the right child node of the current node. Storing encrypted category vectors The variables of the current node Storing encrypted category vectors ; S25: Increment the current node level by 1, and set the left subtree... and right subtree set Each as a new current image vector set Repeat steps S22-S27 until the encrypted image vectors corresponding to all elements in the original image vector set are stored in the variables of the corresponding leaf nodes. In the middle, the encrypted index construction of the tree structure is completed; where, when the left subtree set is... As the new current image vector set When the new current node is the left child of the original current node, the right subtree set is... As the new current image vector set When the current node is selected, it becomes the right child of the original current node.

2. The privacy-preserving cross-modal semantic retrieval method based on a tree structure as described in claim 1, characterized in that, The key is: in, For the key, For symmetric encryption keys, and Both dimensions are Different invertible matrices, The dimension of the image data to be encrypted.

3. The privacy-preserving cross-modal semantic retrieval method based on a tree structure as described in claim 1, characterized in that, In step S23, the current image vector set is encrypted. Each element in The method is as follows: For the current image vector set respectively Each element in Perform the first encryption operation to obtain the encrypted image vector corresponding to each element. The first encryption operation is as follows: elements Expand to dimensional vector ,in, It is a random number. The dimension of the image data to be encrypted; the data owner randomly generates a vector with the main diagonal elements as follows. And the dimension is lower triangular random matrix One of the main diagonal elements is a vector. And the dimension is lower triangular matrix All dimensions used are random matrix Will Encrypt to encrypted image vector .

4. The privacy-preserving cross-modal semantic retrieval method based on a tree structure as described in claim 1, characterized in that, The category vector in step S243 and The encryption method is as follows: Will and Expanded to dimensional vector and ,in It is a random number; the data owner randomly generates a vector with the main diagonal elements. And the dimension is lower triangular random matrix One of the main diagonal elements is a vector. And the dimension is lower triangular matrix One of the main diagonal elements is a vector. And the dimension is lower triangular random matrix One of the main diagonal elements is a vector. And the dimension is lower triangular matrix All dimensions used are random matrix Will and Encrypt into encrypted category vectors respectively and .

5. The privacy-preserving cross-modal semantic retrieval method based on a tree structure as described in claim 1, characterized in that, In step S3, the cloud server queries the encrypted index based on the search token as follows: S31: Determine if the first node in the current priority queue is a leaf node. If it is, set the variable of the current leaf node... Encrypted image vectors stored in Add to result set If not, proceed to step S32; where the initial value of the first node is the root node; S32: Retrieve the variable of the current first node. Storing encrypted category vectors and variables Storing encrypted category vectors And calculate the left trace. and right trace ,in, Functions represent the computation of matrices traces, The encrypted search token; S33: According to and The size update priority queue is as follows: judge and Is the absolute value of the difference between them less than a set threshold? If yes, then add the left and right child nodes of the current first node to the priority queue; if no, then determine... and The size, if Add the left child of the first node to the priority queue if... Add the right child node of the current first node to the priority queue; S34: Use the updated priority queue as the new current priority queue and re-execute steps S31~S33 until the priority queue is empty, then proceed to step S35; S35: Calculate the result sets separately Encrypted image vectors in With search token The trace of the product of the products will be the largest of the previous ones. The encrypted image vector corresponding to each trace As the final query result.

6. The privacy-preserving cross-modal semantic retrieval method based on a tree structure as described in claim 5, characterized in that, Obtain the encrypted image vector under encrypted conditions. With search token The method for Euclidean distance is as follows: The CLIP deep neural network model was used to process the image data. Encoded as dimensional vector ,in, , The number of image data points; using the CLIP deep neural network model, search tokens are generated corresponding to the content of interest, where the search token is... dimensional vector ; Will dimensional vector and If steps S2 to S4 are executed directly, then the encrypted image vector in step S45 will be... With search token The trace of the product is the encrypted image vector. With search token The Euclidean distance between them.

7. The privacy-preserving cross-modal semantic retrieval method based on a tree structure as described in claim 5, characterized in that, Obtain the encrypted image vector under encrypted conditions. With search token The method for cosine similarity is as follows: The CLIP deep neural network model was used to process the image data. Encoded as dimensional vector ,in, , The number of image data points; using the CLIP deep neural network model, search tokens are generated corresponding to the content of interest, where the search token is... dimensional vector ; Will dimensional vector Perform normalization operation ,vector Perform normalization operation Then, steps S2 to S4 are executed, and the encrypted image vector in step S45 is... With search token The trace of the product is the encrypted image vector. With search token Cosine similarity between them.

8. The privacy-preserving cross-modal semantic retrieval method based on a tree structure as described in claim 1, characterized in that, The method for generating search tokens is as follows: Data users generate corresponding search tokens for content of interest using the CLIP deep neural network model, where the search token is... dimensional vector ; The encryption method for the search token is as follows: Search token Expand to dimensional vector ,in, It is a random number. The dimension of the image data to be encrypted; the data user randomly generates a vector with the main diagonal elements as follows. And dimensions For a lower triangular random matrix One of the main diagonal elements is a vector. And the dimension is lower triangular matrix All dimensions used are random matrix Search token Encryption .

Images