A verifiable cross-modal retrieval method

Abstract

The application provides a verifiable cross-modal retrieval method, and belongs to the technical field of Internet of Things security. The method comprises the following steps: a data owner constructs a data set, a 1-norm image vector, a 1-norm text vector, an index table, a prime number table, a ciphertext vector and a Merkle prime number tree, sends the ciphertext vector, the index table and the Merkle prime number tree to a cloud server, and sends a root digest of the Merkle prime number tree, the prime number table, the 1-norm image vector and the 1-norm text vector to a query client; the query client sends a cross-modal query request to the cloud server; the cloud server receives the cross-modal query request, and sends an inner product vector, a group number interval and a verification object set to the query client; and the query client performs integrity verification, correctness verification and Hamming distance calculation according to the inner product vector, the group number interval and the verification object set to find the most similar data. The Merkle prime number tree is constructed by integrating the prime number into the traditional Merkle tree, so that the integrity and correctness of the cross-modal retrieval result are ensured.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Internet of Things security, data privacy protection and cross-modal retrieval, and particularly relates to a verifiable cross-modal retrieval method. BACKGROUND

[0002] With the wide application of intelligent devices and sensors in the Internet of Things, the amount of multi-modal data grows exponentially. In order to reduce the burden of local storage and calculation, a large amount of data is outsourced to the cloud server. However, data outsourcing brings serious privacy leakage risk, especially for cross-modal retrieval tasks involving sensitive information such as text and image.

[0003] In order to protect data privacy, privacy-preserving cross-modal retrieval (PPCMR) technology has gradually become a research hotspot. The core of cross-modal retrieval is to calculate the similarity between feature vectors of different modal data, and vector inner product is one of the key operations for measuring vector similarity. In recent years, an important branch of functional encryption, inner product encryption (IPE), has attracted much attention because it can directly support inner product operations on ciphertext. An ideal IPE scheme allows the data owner to encrypt the vector and generates a key associated with the query vector for the query client; when the key is used to decrypt the ciphertext, the decryption party can only obtain the inner product result of the two vectors, and cannot obtain the original vector information. This provides a powerful cryptographic tool for securely calculating the similarity of cross-modal data (such as Hamming distance which can be converted into inner product form).

[0004] Although the existing PPCMR scheme protects the privacy of data and query requests during the retrieval process, the existing scheme does not have a verification function. In this way, in an untrusted cloud environment, the cloud server may tamper with the cross-modal retrieval result, causing the interests of the query party to be damaged. Specifically, the cloud server may miss the return of the query result, making it impossible to guarantee the integrity of the query; at the same time, the cloud server may return some fake or incorrect query results, making it impossible to guarantee the correctness of the query. Therefore, there is an urgent need for a lightweight cross-modal retrieval scheme that can guarantee data privacy and query security while supporting result verification, in order to promote the practical application of secure outsourcing retrieval in the Internet of Things environment. SUMMARY

[0005] The application aims to provide a verifiable cross-modal retrieval method, a data structure named Merkle Prime Tree (MPT) is designed to ensure the integrity and correctness of cross-modal retrieval results, and a Function-Hiding Inner Product Encryption (FHIPE) method is used to protect the privacy of data and query requests, thereby realizing a lightweight and verifiable cross-modal retrieval scheme in the Internet of Things. Specifically, the application first maps data and query requests into vectors using cross-modal hashing technology; then, the converted vectors are encrypted using function-hiding inner product encryption to protect privacy; then, the characteristics of prime numbers, which can only be divided by themselves and the integer 1, are used to associate prime numbers with data generation time in the Internet of Things, and the MPT structure is designed, so that the client can verify the integrity and authenticity of the returned results. The entire scheme is based on lightweight cryptographic primitives, which can well adapt to the Internet of Things environment while ensuring security. To solve the technical problem that the existing technology protects the privacy of data and query requests during the retrieval process, but the existing schemes do not have verification function.

[0006] To solve the above technical problems, the specific technical solutions of the application are as follows:

[0007] A verifiable cross-modal retrieval method, the method comprising the following steps:

[0008] Step S1: a trusted third party generates public parameters, a master key and a key control key, broadcasts the public parameters to the data owner, the query client and the cloud server, and sends the master key and the key control key to the data owner and the query client;

[0009] Step S2: the data owner constructs a data set of multiple groups of cross-modal images and texts to be stored, and constructs a 1-norm image vector, a 1-norm text vector, an index table, a prime number table, a ciphertext vector, a Merkle prime number tree, an encrypted image and an encrypted text, sends the ciphertext vector, the encrypted image, the encrypted text, the index table and the Merkle prime number tree to the cloud server, and sends the root digest of the Merkle prime number tree, the prime number table, the 1-norm image vector and the 1-norm text vector to the query client;

[0010] Step S3: the cloud server stores the ciphertext vector, the encrypted image, the encrypted text, the index table and the Merkle prime number tree sent by the data owner;

[0011] Step S4: the query client converts the query picture or query text into a query vector, generates a query token, and sends the query token, a query timestamp and a query data type to the cloud server;

[0012] Step S5: The cloud server receives the query request, queries the group number interval according to the query timestamp, constructs a verification object set, calculates an inner product vector, and sends the inner product vector, the group number interval, and the verification object set to the query client;

[0013] Step S6: The query client performs integrity verification, correctness verification, and Hamming distance calculation on the inner product vector, the group number interval corresponding to the query timestamp, and the verification object set to find the most similar data.

[0014] Further, step S2 includes the following steps:

[0015] Step S21: The data owner converts the multiple groups of cross-modal images and texts that need to be stored into image vectors and text vectors through cross-modal hash conversion, and constructs a data set in combination with cross-modal data timestamps;

[0016] Step S22: The data owner calculates the 1-norm of the image vector and the text vector respectively to obtain the 1-norm image vector and the 1-norm text vector;

[0017] Step S23: The data owner divides the cross-modal data into continuous intervals according to the cross-modal data timestamps, and constructs an index table of cross-modal data timestamps-group number intervals;

[0018] Step S24: The data owner constructs a prime number table of cross-modal data timestamps-random prime numbers according to the index table;

[0019] Step S25: The data owner multiplies the text vector with the corresponding prime number in the prime number table according to the cross-modal data timestamp, and replaces the last dimension 0 parameter of the changed text vector with a hash value to obtain an updated text vector; the data owner multiplies the image vector with the corresponding prime number in the prime number table according to the cross-modal data timestamp, and replaces the last dimension 0 parameter of the changed image vector with a hash value to obtain an updated image vector;

[0020] Step S26: The data owner encrypts the updated text vector and the updated image vector respectively to obtain an encrypted text vector and an encrypted image vector, and the encrypted text vector and the encrypted image vector constitute a ciphertext vector;

[0021] Step S27: The data owner constructs a Merkle prime number tree according to the index table and the prime number table;

[0022] Step S28: The data owner encrypts the cross-modal images and texts using symmetric encryption or asymmetric encryption to obtain encrypted images and encrypted texts;

[0023] Step S29: The data owner sends the ciphertext vector, the encrypted image, the encrypted text, the index table and the Merkle prime tree to the cloud server, and sends the root digest of the Merkle prime tree, the prime table, the 1-norm image vector and the 1-norm text vector to the query client.

[0024] Further, step S27 comprises the following steps:

[0025] Step S271: Construct initial node information for each cross-modal data timestamp in the index table;

[0026] Step S272: Construct an array with length to store node information of the Merkle prime tree, the subscript of the array starts from 1, and 1 to stores non-leaf node information of the Merkle prime tree, to stores leaf node information of the Merkle prime tree; is the number of entries of the index table;

[0027] Step S273: For each leaf node , calculate the digest of the leaf node according to the initial node information, and construct node information and store it in the array ;

[0028] Step S274: For non-leaf node , calculate the digest of the non-leaf node in the iteration manner of decreasing to , and construct node information and store it in the array; obtain the Merkle prime tree and the root digest.

[0029] Further, in step S273, the digest of the leaf node is calculated as follows:

[0030]

[0031] wherein, denotes a connection operation; denotes node number; denotes the digest of node ; denotes a hash function; denotes a group number identifier; denotes a cross-modal data timestamp in the index table; denotes a cross-modal data timestamp corresponding prime number in the prime table;

[0032] The node information of the leaf node is , NULL, means no value is filled in; storing node information to an array .

[0033] Further, in step S274, for each non-leaf node , the left child node is numbered , and the right child node is numbered ; according to , the node information stored in the array is found , and according to , the node information stored in the array is found ; the digest of the left child node and the digest of the right child node are hashed to construct the digest of the non-leaf node ; the digest of the non-leaf node is calculated as follows:

[0034]

[0035] wherein represents the digest of the non-leaf node, represents the digest of the left child node of the non-leaf node, represents the digest of the right child node of the non-leaf node;

[0036] The node information of the non-leaf node is constructed , and the node information is stored in the array .

[0037] Further, step S4 includes the following steps:

[0038] Step S41: the query client converts the query picture or query text into a query vector through cross-modal hashing;

[0039] Step S42: the query client generates a query token;

[0040] Step S43: the query client sends the query token, query timestamp, and query data type to the cloud server.

[0041] Further, step S5 includes the following steps:

[0042] Step S51: the cloud server receives the query token, query timestamp, and query data type submitted by the query client, and according to the query timestamp, queries the corresponding group number interval in the index table;

[0043] Step S52: the cloud server extracts the Merkle prime number tree path and sibling nodes corresponding to the query timestamp, and constructs a verification object set;

[0044] Step S53: The cloud server calculates the inner product vector according to the query data type and traverses the ciphertext vector of the corresponding data type in the group number interval;

[0045] Step S54: The cloud server sends the inner product vector, the group number interval corresponding to the query timestamp and the verification object set to the query client.

[0046] Further, step S6 comprises the following steps:

[0047] Step S61: The query client receives the inner product vector, the group number interval corresponding to the query timestamp and the verification object set, and calculates the query node digest of the query node number;

[0048] Step S62: The query client calculates the query root digest according to the digest of the query node and the sibling node in the verification object set, and if the query root digest is the same as the root digest, the integrity verification is passed;

[0049] Step S63: The query client performs correctness verification on the inner product vector;

[0050] Step S64: If the integrity verification and the correctness verification are both passed, the query client calculates the Hamming distance, wherein the minimum Hamming distance is the most similar data;

[0051] Step S65: According to the group number corresponding to the minimum Hamming distance, the client retrieves the data type corresponding to the query data type and the encrypted data of the corresponding group number from the cloud server.

[0052] Compared with the prior art, the present application has the following beneficial technical effects:

[0053] 1) The present application creatively integrates the characteristics of prime numbers that can only be divided by integers 1 and themselves into traditional Merkle trees to construct Merkle prime number trees to ensure the integrity and correctness of cross-modal retrieval results.

[0054] 2) The verification method proposed in the present application only needs some simple hash calculation and division calculation, and has high verification efficiency and strong practicability;

[0055] 3) The present application proposes a secure cross-modal retrieval method supporting verification function, which is based on lightweight cryptography primitives to ensure that it can adapt to the Internet of Things environment. BRIEF DESCRIPTION OF DRAWINGS

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram of the system model for the cross-modal retrieval method of the present invention.

[0058] Figure 2 This is a flowchart illustrating the verifiable cross-modal retrieval method of the present invention.

[0059] Figure 3 This is a schematic diagram illustrating the data encryption of the cross-modal retrieval method of the present invention.

[0060] Figure 4 This is a schematic diagram illustrating the client-side verification of query results for the cross-modal retrieval method of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] This invention proposes a verifiable cross-modal retrieval method, such as... Figure 1 As shown, the system model of the verifiable cross-modal retrieval method includes four types of entities: Trusted Third Party (TP), Data Owner (DO), Query Client (QC), and Cloud Server (CS). Among them, the Trusted Third Party, Data Owner, and Query Client are trusted entities, while the Cloud Server is an untrusted entity. The Cloud Server may return incomplete or incorrect cross-modal retrieval results due to factors such as profit. The functional descriptions of each entity are as follows:

[0063] A trusted third party is responsible for generating global system parameters, generating, managing, and distributing keys.

[0064] The data owner generates multi-modal data of text, images, and is responsible for converting the multi-modal data of text and images into vectors through cross-modal hashing, encrypting the vectors to generate ciphertext vectors by using function-hiding inner product encryption (FHIPE), and outsourcing the ciphertext vectors to the cloud for storage. At the same time, the data owner generates a Merkle prime tree (MPT) and auxiliary information based on the multi-modal data of text and images. In addition, the data owner encrypts the multi-modal data of text, images and the like by using symmetric encryption (such as AES) or asymmetric encryption (such as RSA), and sends the encrypted ciphertext to the cloud for storage.

[0065] The query client converts the query request into a query vector by using cross-modal hashing when retrieval is needed, encrypts the query vector to generate a query token by using its own private key, and submits the query token to the cloud server. When receiving the retrieval result and the verifiable object set from the cloud server After that, the integrity and correctness verification is completed locally.

[0066] The cloud server is responsible for storing the massive encrypted data uploaded by the data owner. When receiving the query token submitted by the query client, the cloud server directly performs cross-modal retrieval operation on the ciphertext data, and returns the obtained retrieval result and the verifiable object set to the query client.

[0067] The verifiable cross-modal retrieval method provided by the application includes the following steps, as shown in Figure 2

[0068] Step S1: The trusted third party generates public parameters, a master key and a key control key, broadcasts the public parameters to the data owner, the query client and the cloud server, and sends the master key and the key control key to the data owner and the query client.

[0069] Specifically, the trusted third party calls the initialization method of function-hiding inner product encryption to generate public parameters and a master key, which are represented as follows:

[0070]

[0071] Among them, represents the initialization method of function-hiding inner product encryption; represents the public parameter; represents the master key; represents a security parameter; represents a dimension set, , represents a ring of modulus ; and .​ denotes modulus; dimension set is a modulus ring from which a finite set is randomly selected.

[0072] security parameter and modulus are positive integers, which are reasonably set as needed.

[0073] A trusted third party randomly generates a string of length as a key control key .

[0074] The trusted third party broadcasts the generated public parameters to the data owner, query client and cloud server, and sends the master key and key control key to the data owner and query client.

[0075] Step S2: The data owner constructs a data set of multiple groups of cross-modal image and text to be stored, and constructs 1-norm image vector, 1-norm text vector, index table, prime table, ciphertext vector, Merkle prime tree, encrypted image and encrypted text, sends the ciphertext vector, encrypted image, encrypted text, index table and Merkle prime tree to the cloud server, and sends the root digest of the Merkle prime tree, prime table, 1-norm image vector and 1-norm text vector to the query client.

[0076] Step S21: The data owner converts the multiple groups of cross-modal image and text to be stored into image vector and text vector through cross-modal hash conversion, and constructs a data set in combination with cross-modal data timestamp.

[0077] The data owner converts the multiple groups of cross-modal image and text to be stored into image vector and text vector through cross-modal hash conversion, and the group image is represented as , the group text is represented as , the group hash image vector and the group hash text vector are obtained through cross-modal hash conversion; and a dimension 0 parameter is finally added to each group of hash image vector and hash text vector to obtain image vector and text vector .

[0078] The data owner constructs a data set by using image vector, text vector and timestamp, and the constructed data set is represented as ; wherein represents the group of cross-modal data; represents the group of cross-modal data timestamp, generating time for cross-modal image and text; represents the cross-modal data group number; represents the total amount of cross-modal data. The data in the data set is arranged in ascending order according to the cross-modal data timestamp .

[0079] Step S22: The data owner calculates the 1-norm of the image vector and the text vector respectively, and obtains the 1-norm image vector and the 1-norm text vector.

[0080] For each group of image vectors and text vectors , the 1-norm is calculated respectively, and the 1-norm of the image vector and the 1-norm of the text vector are obtained, and all the 1-norm of the image vector and the 1-norm of the text vector constitute the 1-norm image vector and the 1-norm text vector .

[0081] Step S23: The data owner divides the cross-modal data into continuous intervals according to the cross-modal data timestamp, and constructs an index table of cross-modal data timestamp-group number interval.

[0082] The data owner classifies the same cross-modal data timestamp, and allocates continuous intervals according to the cross-modal data group number, and constructs an index table of cross-modal data timestamp-group number interval . If the timestamps of the group of cross-modal data to the group of cross-modal data are all , then the cross-modal data timestamp of the corresponding index table is , and the group number interval is .

[0083] Step S24: The data owner constructs a prime table of cross-modal data timestamp-random prime number according to the index table.

[0084] The data owner randomly selects a non-repeated prime number from the prime number set for the cross-modal data timestamp of each record in the index table , and constructs a prime table of cross-modal data timestamp-random prime number .

[0085] Step S25: The data owner generates the updated text vector according to the cross-modal data timestamp and replaces the last dimension 0 of the changed text vector with the hash value to obtain the updated text vector Step S24: The data owner generates the updated image vector according to the cross-modal data timestamp and replaces the last dimension 0 of the changed image vector with the hash value to obtain the updated image vector .

[0086] The hash value is generated through a key and the cross-modal data group number corresponding to the text vector / image vector , and is expressed as follows:

[0087]

[0088] wherein, represents the hash value of the th group; represents the hash function.

[0089] Step S26: The data owner encrypts the updated text vector and the updated image vector respectively to obtain the encrypted text vector and the encrypted image vector, and the encrypted text vector and the encrypted image vector constitute the ciphertext vector.

[0090] The data owner calls the encryption algorithm of the function-hiding inner product encryption to generate the encrypted text vector from the updated text vector, and is expressed as follows:

[0091]

[0092] wherein, represents the encryption algorithm of the function-hiding inner product encryption; represents the th group of encrypted text vectors, represents the master key, represents the updated text vector.

[0093] The data owner calls the encryption method of the function-hiding inner product encryption to generate the encrypted image vector from the updated image vector, and is expressed as follows:

[0094]

[0095] wherein, represents the th group of encrypted image vectors, represents the master key, represents the updated image vector.

[0096] The encrypted text vector and the encrypted image vector together form the ciphertext vector representation as follows: .

[0097] Step S27: The data owner constructs a Merkle Prime Tree (MPT) based on the index table and prime number table, using symbolic... express.

[0098] The Merkle prime tree (MPT) is a full binary tree. The root node is numbered 1, and each timestamp in the index table corresponds to a leaf node, meaning the number of leaf nodes is the same as the number of entries in the index table. The MPT is constructed from bottom to top and from right to left, as follows:

[0099] Step S271: Construct initial node information for each cross-modal data timestamp in the index table.

[0100] The data owner constructs initial node information for each cross-modal data timestamp in the index table. The initial node information is represented as follows:

[0101]

[0102] in, Indicates the first Initial node information; Indicates the initial node label. , This indicates the number of initial node information entries, corresponding to the number of entries in the index table. This represents the group number identifier, i.e., the timestamp of cross-modal data in the index table. All group numbers within the corresponding group number range; This represents the cross-modal data timestamp corresponding to the initial node; Represents the timestamps of cross-modal data in the prime number table. The corresponding prime number.

[0103] Step S272: Construct a length of array An array that stores the node information of a Merkle prime tree. The subscripts start from 1, from 1 to Stores information about non-leaf nodes in the Merkle prime tree. to Stores information about the leaf nodes in the Merkle prime tree.

[0104] The number of leaf nodes in a Merkle prime tree is equal to the number of initial node information. According to the properties of a full binary tree, the total number of nodes in a Merkle prime tree is . Therefore, the present invention uses a length of array to store the node information of the Merkle prime tree, array whose subscript starts from 1, 1 to to store the node information of the non-leaf nodes in the Merkle prime tree, to to store the node information of the leaf nodes in the Merkle prime tree.

[0105] Step S273: for each leaf node , according to the initial node information, perform hash calculation to construct the digest of the leaf node ; and construct the node information and store it to array .

[0106] The calculation method of the digest of the leaf node is as follows:

[0107]

[0108] wherein, represents the digest of the leaf node.

[0109] Design a quintuple information to represent the node information of the node , and the quintuple information includes node number, left child node number, right child node number, digest of the node, and time stamp.

[0110] The node information of the leaf node does not have left child node number and right child node number, and the node information of the leaf node is constructed as , which represents null and is not filled with value; which represents node number; and the node information is stored to array .

[0111] Step S274: for the non-leaf node , calculate the digest of the non-leaf node in the iteration manner from decreasing to , and construct the node information and store it to array ; obtain the Merkle prime tree and the root digest .

[0112] For the non-leaf node, the construction is performed in the iteration manner from decreasing to . Specifically, for each non-leaf node , the left child node number thereof is , and the right child node number thereof is . According to , find the node information stored in array . , similarly, according to , find the node information stored in the array . , the digest of the left child node is hashed with the digest of the right child node to construct the digest of the non-leaf node , the digest of the non-leaf node is calculated as follows:

[0113]

[0114] wherein, represents the digest of the non-leaf node, represents the digest of the left child node of the non-leaf node, represents the digest of the right child node of the non-leaf node.

[0115] The node information of the non-leaf node has no timestamp, and the node information of the non-leaf node is constructed , the node information is stored in the array .

[0116] Then decrement 1, calculate the digest of the non-leaf node and construct the node information. When decreases to 0, the iteration process is terminated, and the Merkle prime tree is constructed, and the root digest is also represented by the symbol .

[0117] Step S28: The data owner encrypts the cross-modal image and text using symmetric encryption (such as AES) or asymmetric encryption (such as RSA) to obtain encrypted images and encrypted text.

[0118] Step S29: The data owner sends the ciphertext vector , encrypted images, encrypted text, index table and Merkle prime tree to the cloud server, and sends the root digest of the Merkle prime tree , prime table , 1-norm image vector and 1-norm text vector to the query client.

[0119] Step S3: The cloud server stores the ciphertext vector , encrypted images, encrypted text, index table and Merkle prime tree sent by the data owner.

[0120] Step S4: The query client converts the query picture or query text into a query vector, generates a query token, and sends the query token, query timestamp, and query data type to the cloud server.

[0121] Step S41: The query client converts the query picture or query text into a query vector through cross-modal hashing.

[0122] The cross-modal retrieval can retrieve through picture query text or text query picture. The query client generates a random number The hash query vector is increased by one dimension parameter set as the random number to obtain the query vector .

[0123] Step S42: The query client generates a query token.

[0124] The query client calls the key generation algorithm of the function hidden inner product encryption to generate the query token, which is represented as follows:

[0125]

[0126] wherein, denotes the key generation algorithm of the function hidden inner product encryption; denotes the query token, denotes the master key.

[0127] Step S43: The query client sends the query token, query timestamp, and query data type to the cloud server.

[0128] The query timestamp is the generation time of the query picture and the query text, and the query client sends the query token , the query timestamp , and the query data type to the cloud server.

[0129] Step S5: The cloud server receives the query request, queries the group number interval according to the query timestamp, constructs a verification object set, calculates an inner product vector, and sends the inner product vector, group number interval, and verification object set to the query client.

[0130] Step S51: The cloud server receives the query token , query timestamp , and query data type submitted by the query client, queries the corresponding group number interval in the index table according to the query timestamp.

[0131] Step S52: The cloud server extracts the Merkle prime tree path and sibling nodes corresponding to the query timestamp and constructs a set of verification objects.

[0132] Cloud servers based on query timestamps Find the Merkel prime tree The corresponding leaf node And record the corresponding query node number. Query node number Merkel prime tree From the root node to the leaf node The sibling nodes on the path constitute the set of verification objects. .

[0133] Step S53: The cloud server iterates through the ciphertext vectors of the corresponding data types within the group number range according to the query data type and calculates the inner product vector.

[0134] If the group number range is For the ciphertext vector of each cross-modal data set within the group number range, if the query data type... For images, the cloud server uses the function to call encrypted image vectors to hide the inner product encryption decryption algorithm to calculate the inner product value, as shown below:

[0135]

[0136] in, This indicates the decryption algorithm for feature-hidden inner product encryption; The first element of the inner product vector is... Each inner product value, Indicates common parameters, Indicates a query token. This represents an encrypted image vector.

[0137] If the query data type For text, the cloud server uses the function to call the encrypted text vector to hide the inner product encryption decryption algorithm to calculate the inner product value, as shown below:

[0138]

[0139] in, This represents an encrypted text vector.

[0140] Step S54: The cloud server will convert the inner product vector Query the group number range and verification object set corresponding to the timestamp. Send to the query client.

[0141] Step S6: The query client performs integrity verification, correctness verification, and Hamming distance calculation based on the inner product vector, the group number range corresponding to the query timestamp, and the set of verification objects to find the most similar data.

[0142] Step S61: Query the client to receive the inner product vector Query the group number range and verification object set corresponding to the timestamp. Calculate the query node number The query node summary.

[0143] The query client uses the query timestamp. Query the prime number table for the prime number corresponding to the query timestamp, obtain the group number identifier based on the group number range, and calculate the summary of the query node using the summary calculation method of the leaf node. .

[0144] Step S62: The query client calculates the query root digest based on the query node's digest and the sibling nodes in the verification object set. If the query root digest is the same as the root digest, the integrity verification passes.

[0145] The query client combines the summary of the query node with the set of verification objects. The digests of the sibling nodes of the query node are concatenated and hashed again to generate the digest of the parent node. The digest of the parent node is then combined with the set of verification objects. The digests of the sibling nodes of the parent node are concatenated and then hashed again to generate the digest of the parent node. This process is repeated until the root query digest is obtained. Compare query root summaries With root summary If they are the same, the integrity verification passes, and the next step of correctness verification is performed.

[0146] Step S63: Query the client to verify the correctness of the inner product vector.

[0147] This invention utilizes the property that prime numbers can only be factored by themselves and the integer 1, to update the inner product vector. Divide by the timestamp in the prime number table The corresponding prime number is used to verify the correctness of the returned result based on whether it is an integer.

[0148] The query client first removes the random value from the inner product result, that is, for Each inner product value The query client updates it to Next, the updated version will be available. Divide by the timestamp in the prime number table If the corresponding prime numbers are all integers, it proves that the cloud server did not use other vectors or fake vectors to calculate the inner product value, and the correctness verification is passed.

[0149] Step S64: If both integrity and correctness verifications pass, query the client to calculate the Hamming distance, where the data with the smallest Hamming distance is the most similar data.

[0150] For each inner product value If the query data type For images, the query client uses the 1-norm image vector. The Hamming distance is calculated as follows:

[0151]

[0152] in, Indicates the first Hamming distance, Indicates the first Group 1 norm image.

[0153] If the query data type For text, the query client uses a text vector with a 1-norm. The Hamming distance is calculated as follows:

[0154]

[0155] in, Indicates the first Group 1 norm text.

[0156] Step S65: Based on the group number corresponding to the minimum Hamming distance The client retrieves data of type [type] from the cloud server. Group number is Encrypted data.

[0157] Example 1

[0158] The following example, a cross-modal retrieval of "image-to-text retrieval," illustrates in detail the execution steps of the method proposed in this invention. In the example, assume that the data owner holds a dataset containing 5 text data entries. Each text data entry is converted into a 6-dimensional feature vector through cross-modal hashing, with the 7th dimension initialized to 0 to hold a random value.

[0159] like Figures 3-4 As shown, the verifiable cross-modal retrieval method includes the following steps:

[0160] Step S1: A trusted third party generates public parameters, a master key, and a keying key. The public parameters are broadcast to the data owner, the query client, and the cloud server. The master key and keying key are sent to the data owner and the query client.

[0161] The trusted third-party call function hides the initialization method of inner product encryption, generating public parameters and the master key, as shown below:

[0162]

[0163] in, Indicates common parameters; Indicates the master key; Indicates safety parameters; Represents a set of dimensions.

[0164] Step S2: The data owner constructs a dataset from multiple sets of cross-modal texts that need to be stored, and builds a 1-norm text vector, an index table, a prime number table, a ciphertext vector, a Merkle prime tree, and encrypted text. The ciphertext vector, encrypted text, index table, and Merkle prime tree are sent to the cloud server, and the root digest of the Merkle prime tree, the prime number table, and the 1-norm text vector are sent to the query client.

[0165] This embodiment uses image cross-modal text retrieval as an example. For convenience, the explanation will focus on storing only cross-modal text data.

[0166] Step S21: The data owner converts the multiple sets of cross-modal texts that need to be stored into text vectors through cross-modal hashing, and combines them with cross-modal data timestamps to construct a dataset.

[0167] The data owner converts multiple sets of cross-modal text that need to be stored into hash text vectors using cross-modal hashing. Group hash text vector representation is In each set of hash text vectors Finally, add a one-dimensional zero-parameter vector to obtain the text vector. .

[0168] The data owner constructs a dataset using text vectors and timestamps; the constructed dataset is represented as follows: ;in, Indicates the first Group cross-modal data; Indicates the first Group cross-modal data timestamps; Indicates the cross-modal data group number; This indicates the total amount of data across modules. (Dataset) Data in the data is categorized by cross-modal data timestamps. Sort in ascending order.

[0169] Step S22: The data owner calculates the 1-norm of the text vector, obtaining the 1-norm text vector.

[0170] For each group of text vectors Calculate the 1-norm, obtaining the 1-norm of the text vector The 1-norm of all text vectors constitutes the 1-norm text vector .

[0171] Step S23: The data owner divides the cross-modal data into consecutive intervals according to the cross-modal data timestamp, and constructs an index table of cross-modal data timestamp-group number intervals.

[0172] The data owner classifies the same cross-modal data timestamp, and assigns consecutive intervals according to the cross-modal data group number, and constructs an index table of cross-modal data timestamp-group number intervals . If the timestamp of the first group of cross-modal data to the first group of cross-modal data is the same as , then the cross-modal data timestamp of the corresponding index table is , and the group number interval is .

[0173] Step S24: The data owner constructs a prime number table of cross-modal data timestamp-random prime number according to the index table.

[0174] The data owner randomly selects a non-repeating prime number from the prime number set for the cross-modal data timestamp of each record in the index table , and constructs a prime number table of cross-modal data timestamp-random prime number .

[0175] Step S25: The data owner multiplies the text vector with the corresponding prime number in the prime number table according to the cross-modal data timestamp, and replaces the last dimension 0 parameter of the text vector with a hash value, obtaining the updated text vector .

[0176] For example, the text vector multiplied by the corresponding prime number 23 becomes ; the hash value is appended to the last dimension of the vector, for example, if , then the updated text vector is .

[0177] Step S26: The data owner encrypts the updated text vector to obtain an encrypted text vector and an encrypted text vector constitutes a ciphertext vector.

[0178]

[0179] The encrypted text vector and the encrypted image vector constitute a ciphertext vector, which is expressed as: .

[0180] Step S27: The data owner constructs a Merkle prime tree according to the index table and the prime table .

[0181] Step S271: For each cross-modal data timestamp in the index table, initial node information is constructed.

[0182] The data owner constructs initial node information for each cross-modal data timestamp in the index table, and the initial node information is expressed as follows:

[0183]

[0184] wherein, represents the i-th initial node information; represents an initial node label, ; represents a group number identifier, that is, all group numbers within the group number interval corresponding to the cross-modal data timestamp in the index table . represents the cross-modal data timestamp corresponding to the initial node; represents the prime number in the prime table corresponding to the cross-modal data timestamp .

[0185] Step S272: Step S272: An array with a length of 7 is constructed to store the node information of the Merkle prime tree, and the subscript of the array starts from 1, and 1 to 3 store the non-leaf node information in the Merkle prime tree, and 4 to 7 store the leaf node information in the Merkle prime tree.

[0186] Step S273: For each leaf node , the initial node information is hashed to construct the digest of the leaf node , and the node information is stored in the array .

[0187] Step S274: For non-leaf nodes , the digest of the non-leaf node is calculated in a descending manner from to , and the node information is stored in the array​ ; obtain merkle prime tree and root digest .

[0188] Step S28: the data owner encrypts the cross-modal text by using symmetric encryption (such as AES) or asymmetric encryption (such as RSA), to obtain encrypted text.

[0189] Step S29: the data owner sends the ciphertext vector , the encrypted text, the index table and the merkle prime tree to the cloud server, and sends the root digest of the merkle prime tree, the prime table and the 1-norm text vector to the query client.

[0190] Step S3: the cloud server stores the ciphertext vector , the encrypted text, the index table and the merkle prime tree sent by the data owner.

[0191] Step S4: the query client converts the query picture into a query vector, generates a query token, and sends the query token, the query timestamp and the query data type to the cloud server.

[0192] Step S41: the query client converts the query picture into a query vector by cross-modal hashing.

[0193] The query client converts the query picture or the query text into a hash query vector by cross-modal hashing; the query client generates a random number , and adds a one-dimensional parameter of the hash query vector to the random number to obtain a query vector .

[0194] Step S42: the query client calls the key generation algorithm of the function hidden inner product encryption to generate a query token, which is expressed as follows:

[0195]

[0196] wherein, denotes the query token, denotes the master key.

[0197] Step S43: the query client sends the query token , the query timestamp and the query data type to the cloud server.

[0198] Step S5: The cloud server receives the query request, queries the group number range based on the query timestamp, constructs the set of verification objects, calculates the inner product vector, and sends the inner product vector, group number range, and set of verification objects to the query client.

[0199] Step S51: The cloud server receives the query token submitted by the query client. Query timestamp Query data types Based on the query timestamp, retrieve the corresponding group number range from the index table. .

[0200] Step S52: The cloud server extracts the Merkle prime tree path and sibling nodes corresponding to the query timestamp and constructs a set of verification objects.

[0201] Cloud servers based on query timestamps Find the Merkel prime tree The corresponding leaf node And record the corresponding query node number. Query node number Merkel prime tree From the root node to the leaf node The sibling nodes on the path constitute the set of verification objects. .

[0202] Step S53: The cloud server iterates through the ciphertext vectors of the corresponding data types within the group number range according to the query data type and calculates the inner product vector.

[0203] For group number range The ciphertext vector of each cross-modal data in Based on the query data type being text, the cloud server invokes the hidden inner product encryption decryption algorithm to calculate the inner product value, obtains the inner product value, and stores it. ,Right now , .

[0204] Step S54: The cloud server will convert the inner product vector Query the group number range and verification object set corresponding to the timestamp. Send to the query client.

[0205] Step S6: The query client performs integrity verification, correctness verification, and Hamming distance calculation based on the inner product vector, the group number range corresponding to the query timestamp, and the set of verification objects to find the most similar data.

[0206] Step S61: Query the client to receive the inner product vector Query the group number range and verification object set corresponding to the timestamp. , the query node number is calculated the query node digest.

[0207] the digest of the computing node .

[0208] Step S62: The query client calculates the query root digest according to the query node digest and the sibling nodes in the verification object set.

[0209] According to and in , the digest of node 2 is calculated by hash calculation ; then according to and in , the query root digest is calculated by hash calculation , satisfying , and the integrity verification is passed.

[0210] Step S63: The query client performs correctness verification on the inner product vector.

[0211] The query client first removes the random value from the inner product result, that is, the query client updates it to , . Then divide the updated by the prime number to obtain the result, which is an integer, so the correctness verification is passed.

[0212] Step S64: If the integrity verification and the correctness verification are both passed, the query client calculates the Hamming distance, wherein the smallest Hamming distance is the most similar data.

[0213] The query client calculates the Hamming distance according to the formula , and obtains . Since , the text vector is the most similar to the query vector .

[0214] Step S65: According to the Hamming distance, the query client retrieves the encrypted text form of the text from the cloud server.

[0215] ​It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A verifiable cross-modal retrieval method, characterized in that, The method includes the following steps: Step S1: A trusted third party generates public parameters, a master key, and a keying key, broadcasts the public parameters to the data owner, the query client, and the cloud server, and sends the master key and keying key to the data owner and the query client. Step S2: The data owner constructs a dataset from multiple sets of cross-modal images and texts that need to be stored, and builds a 1-norm image vector, a 1-norm text vector, an index table, a prime number table, a ciphertext vector, a Merkle prime tree, encrypted images, and encrypted text. The ciphertext vector, encrypted images, encrypted text, index table, and Merkle prime tree are sent to the cloud server, and the root summary of the Merkle prime tree, the prime number table, the 1-norm image vector, and the 1-norm text vector are sent to the query client. Step S3: The cloud server stores the ciphertext vector, encrypted image, encrypted text, index table, and Merkle prime tree sent by the data owner; Step S4: The query client converts the query image or query text into a query vector, generates a query token, and sends the query token, query timestamp, and query data type to the cloud server; Step S5: The cloud server receives the query request, queries the group number range based on the query timestamp, constructs the set of verification objects, calculates the inner product vector, and sends the inner product vector, group number range, and set of verification objects to the query client. Step S6: The query client performs integrity verification, correctness verification, and Hamming distance calculation based on the inner product vector, the group number range corresponding to the query timestamp, and the set of verification objects to find the most similar data; Step S2 includes the following steps: Step S21: The data owner converts the multiple sets of cross-modal images and texts that need to be stored into image vectors and text vectors through cross-modal hashing, and constructs a dataset by combining the cross-modal data timestamps; Step S22: The data owner calculates the 1-norm of the image vector and the text vector respectively to obtain the 1-norm image vector and the 1-norm text vector; Step S23: The data owner divides the cross-modal data into continuous intervals according to the cross-modal data timestamp and constructs an index table of cross-modal data timestamp-group number intervals; Step S24: The data owner constructs a prime number table of cross-modal data timestamps-random primes based on the index table; Step S25: The data owner multiplies the text vector with the corresponding prime number in the prime number table based on the cross-modal data timestamp, and replaces the last dimension 0 parameter of the changed text vector with a hash value to obtain the updated text vector; The data owner multiplies the image vector with the corresponding prime number in the prime number table based on the cross-modal data timestamp, and replaces the last dimension 0 parameter of the changed image vector with a hash value to obtain the updated image vector; Step S26: The data owner encrypts the updated text vector and the updated image vector respectively to obtain the encrypted text vector and the encrypted image vector. The encrypted text vector and the encrypted image vector constitute the ciphertext vector. Step S27: The data owner constructs a Merkle prime tree based on the index table and the prime number table; Step S28: The data owner uses symmetric or asymmetric encryption to encrypt the cross-modal image and text, resulting in encrypted image and encrypted text; Step S29: The data owner sends the ciphertext vector, encrypted image, encrypted text, index table, and Merkle prime tree to the cloud server, and sends the root digest of the Merkle prime tree, prime table, 1-norm image vector, and 1-norm text vector to the query client.

2. The verifiable cross-modal retrieval method according to claim 1, characterized in that, Step S27 includes the following steps: Step S271: Construct initial node information for each cross-modal data timestamp in the index table; Step S272: Construct a length of array An array that stores the node information of a Merkle prime tree. The subscripts start from 1, from 1 to Stores information about non-leaf nodes in the Merkle prime tree. to Stores information about the leaf nodes in the Merkle prime tree; This represents the number of entries in the index table. Step S273: For each leaf node The leaf node digest is constructed by performing hash calculation based on the initial node information; And construct node information and store it in an array. ; Step S274: For non-leaf nodes ,according to Decrease to The summaries of non-leaf nodes are calculated using an iterative method, and the node information is constructed and stored in an array; thus, the Merkle prime tree and the root summary are obtained.

3. The verifiable cross-modal retrieval method according to claim 2, characterized in that, In step S273, the summary of the leaf nodes is calculated as follows: in, Indicates a connection operation; Indicates the node number; Represents a node Abstract; Represents a hash function; Indicates the group number identifier; Indicates the timestamps of cross-modal data in the index table; Represents the timestamps of cross-modal data in the prime number table. The corresponding prime number; The node information for constructing leaf nodes is: , Indicates empty, no value to be filled; will store node information. Store to array .

4. The verifiable cross-modal retrieval method according to claim 3, characterized in that, In step S274, for each non-leaf node Its left child node number is Its right child node number ;according to Find the array Stored node information Similarly, according to Find the array Stored node information The digest of a non-leaf node is constructed by hashing the digests of its left and right children. The summary calculation method for non-leaf nodes is as follows: in, This represents a summary of non-leaf nodes. This represents a summary of the left child nodes of a non-leaf node. This represents a summary of the right child node of a non-leaf node. Construct node information for non-leaf nodes , node information Store to array .

5. The verifiable cross-modal retrieval method according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: The query client converts the query image or query text into a query vector using cross-modal hashing; Step S42: The query client generates a query token; Step S43: The query client sends the query token, query timestamp, and query data type to the cloud server.

6. The verifiable cross-modal retrieval method according to claim 1, characterized in that, Step S5 includes the following steps: Step S51: The cloud server receives the query token, query timestamp, and query data type submitted by the query client, and queries the corresponding group number range in the index table based on the query timestamp; Step S52: The cloud server extracts the Merkle prime tree path and sibling nodes corresponding to the query timestamp, and constructs a set of verification objects; Step S53: The cloud server iterates through the ciphertext vectors of the corresponding data types within the group number range according to the query data type and calculates the inner product vector; Step S54: The cloud server sends the inner product vector, the group number range corresponding to the query timestamp, and the set of verification objects to the query client.

7. The verifiable cross-modal retrieval method according to claim 1, characterized in that, Step S6 includes the following steps: Step S61: The query client receives the inner product vector, the group number range corresponding to the query timestamp, and the set of verification objects, and calculates the query node summary of the query node number; Step S62: The query client calculates the query root digest based on the digest of the query node and the sibling nodes in the verification object set. If the query root digest is the same as the root digest, the integrity verification passes. Step S63: Query the client to verify the correctness of the inner product vector; Step S64: If both integrity and correctness verifications pass, query the client to calculate the Hamming distance, where the data with the smallest Hamming distance is the most similar data; Step S65: Based on the group number corresponding to the minimum Hamming distance, the client retrieves the data type corresponding to the query data type and the encrypted data corresponding to the group number from the cloud server.

Images