A secure and verifiable cross-modal retrieval method

Abstract

The present application relates to the technical field of information retrieval privacy protection, and discloses a kind of safe verifiable cross-modal retrieval method, including data owner based on the high-dimensional vector corresponding to image modal and text modal in multimodal data set, corresponding image verifiable B+ tree and text verifiable B+ tree are constructed, and after encryption Multimodal data, high-dimensional vector and root node signature value after digital signature are stored to data service provider, for subsequent receiving query vector generated by query user, cross-modal retrieval is carried out.The present application can quickly judge the distance between node and query vector during retrieval by constructing verifiable B+ tree containing node data, index key value, center point, radius and abstract, unnecessary subtree traversal is avoided, and the query efficiency is improved;And result verification and signature verification further guarantee the authenticity of query result.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, and in particular to a secure and verifiable cross-modal retrieval method. Background Technology

[0002] In many applications, data can exist in multiple modalities. For example, in addition to image content, there is textual information, such as image tags on Flickr and many other social networking sites; this type of data is often referred to as multimodal data. Therefore, cross-modal retrieval technology is needed to effectively organize and manage this massive, heterogeneous multimodal data, uncovering potential correlations between different modalities, helping users quickly and accurately find valuable information from complex data, and avoiding the frustration of information overload. Specifically, cross-modal retrieval aims to retrieve semantically relevant samples from one modality using a query request from another. Specifically, cross-modal retrieval can query images using text or text using images; for example, in the medical field, cross-modal retrieval allows users to find semantically relevant medical images using text descriptions.

[0003] To alleviate the demands of massive storage, computing, and communication, large-scale multimodal data is often outsourced to cloud platforms. However, cloud platforms store vast amounts of data from various users, posing risks of malicious tampering, leakage, or corruption. Furthermore, in the case of outsourced multimodal data, data needs to be transferred between users and the cloud platform; the instability of the network environment can lead to packet loss, errors, or delays during data transmission. Therefore, this raises significant concerns about the authenticity of search results. Result authenticity includes completeness and correctness. Completeness is typically ensured through authenticated query schemes, while correctness requires that all returned objects originate from the data owner and have not been altered.

[0004] Existing authentication query schemes can only handle single-modal queries and cannot be directly applied to cross-modal scenarios; while traditional cross-modal retrieval cannot perform verification and cannot determine whether the returned object has been modified; therefore, existing technologies cannot simultaneously guarantee the authenticity of query results and query efficiency in cross-modal retrieval. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the existing technology cannot simultaneously guarantee the authenticity of search results and the efficiency of search.

[0006] To address the aforementioned technical problems, this invention provides a secure and verifiable cross-modal retrieval method, comprising:

[0007] The data owner extracts features from the text and image modalities of the multimodal data in the multimodal dataset, obtains high-dimensional vectors for the corresponding modalities, and constructs verifiable B+ trees for the corresponding modalities, including:

[0008] Cluster high-dimensional vectors, use the clusters as leaf nodes, and construct the internal nodes and root node of the B+ tree based on the range of index key values ​​of all high-dimensional vectors in all leaf nodes to form the basic B+ tree;

[0009] Calculate the center point and radius of each node based on its membership relationship;

[0010] Using a hash function, a hash chain is generated in a bottom-up manner, and the digest of each node is calculated;

[0011] The center point, radius, summary, and index key values ​​of all high-dimensional vectors of the nodes are stored in the base B+ tree as a verifiable B+ tree.

[0012] Digitally sign the root node of the verifiable B+ tree for each modality and obtain the corresponding root node signature value;

[0013] Encrypt each multimodal data point in the multimodal dataset to obtain encrypted multimodal data; digitally sign the high-dimensional vector corresponding to each encrypted multimodal data point to obtain the digital signature value of the corresponding modality;

[0014] The verifiable B+ tree, root node signature value, all encrypted multimodal data, and digital signature value will be uploaded to the data service provider for storage.

[0015] The data service provider obtains query vectors generated by users based on different modalities; based on the modality of the query vector, it traverses the verifiable B+ tree corresponding to another modality:

[0016] If the current traversal point is an internal node, then traverse its child nodes;

[0017] If the current traversal point is a leaf node, calculate the distance between the center point of the leaf node and the query vector, obtain the k vectors with the smallest distance, and construct the query result set; based on the query result set, the center point, radius, summary of the node, and the signature value of the root node, generate a set of verification objects for the query result set so that the query user can obtain the verified query result set and obtain the real retrieval result of the query vector.

[0018] Preferably, based on the membership relationship of each node, the center point and radius of each node are calculated, including:

[0019] Calculate the center point of the leaf node based on all the high-dimensional vectors in the cluster represented by the leaf node. Represented as: m represents the total number of high-dimensional vectors in the cluster represented by the leaf node L, V i Let L represent the i-th high-dimensional vector in the cluster represented by the leaf node L, where 1 ≤ i ≤ m;

[0020] The radius r of a leaf node is obtained based on the distance between the center point of the leaf node and all high-dimensional vectors in the cluster represented by the leaf node. L , is represented as: S represents the set of all high-dimensional vectors in the clusters represented by the leaf nodes. Represents the center point of the leaf node The distance between the i-th high-dimensional vector in the cluster represented by the leaf node;

[0021] Calculate the geometric center of the center points of all child nodes of an internal node, and use that center point as the center point of the internal node. Represented as: k represents the total number of child nodes of the internal node N. Let represent the center point of the j-th child node of the internal node, where 1 ≤ j ≤ k;

[0022] Calculate the minimum hypersphere radius that can cover all child nodes of the internal node, and use it as the radius r of that internal node. N , is represented as: Indicates the center point of the internal node With the j-th child node N of the internal node j The distance between them Represents the radius of the j-th child node of an internal node;

[0023] Calculate the geometric center of the center point of all child nodes of the root node, and use that center point as the center point of the root node.

[0024] Calculate the minimum hypersphere radius that can cover all child nodes of the root node, and use it as the radius r of the root node. root .

[0025] Preferably, a hash chain is generated in a bottom-up manner using a hash function, and a digest of each node is calculated, including:

[0026] Calculate the summary d of leaf node L L , is represented as:

[0027]

[0028] Calculate the summary d of internal node N N , is represented as:

[0029]

[0030] Calculate the root node's summary d root , is represented as:

[0031]

[0032] Where h() represents a hash function, (id L |K L |V i ) represents the <identifier, index key value\high-dimensional vector> of a leaf node; K1 to K j Let represent the index key values ​​of the 1st to jth child nodes of the internal node N, respectively. to Let K′1 and K′j represent the summaries of the 1st to jth child nodes of the internal node N, respectively. n These represent the index key values ​​of the 1st to nth child nodes of the root node, respectively. to These represent the summaries of the 1st to nth child nodes of the root node, where n represents the total number of child nodes of the root node.

[0033] Preferably, calculating the index key value of a high-dimensional vector includes:

[0034] Based on the cluster center point O in the i-th cluster i Dist(O) to the high-dimensional vector p i ,p), calculate the index key K of the high-dimensional vector. p , is represented as:

[0035] K p =i*ρ+dist(O i ,p);

[0036] Where ρ is the preset partitioning parameter.

[0037] Preferably, obtaining the high-dimensional vectors corresponding to the text modality and image modality in each multimodal data includes:

[0038] Image data from multimodal data is input into an image modality deep neural network to obtain high-dimensional image vectors corresponding to the multimodal data.

[0039] The bag-of-words model is used to convert text data in multimodal data into bag-of-words vectors, and the bag-of-words vectors are input into a text modality deep neural network to obtain high-dimensional text vectors corresponding to the multimodal data.

[0040] Preferably, the image modality deep neural network is a convolutional neural network, comprising, in sequence: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer.

[0041] Preferably, the text modality deep neural network includes, in series, a first fully connected layer with ReLU activation function and a second fully connected layer with identity function activation function.

[0042] Preferably, based on the modality of the query vector, traversing the verifiable B+ tree corresponding to another modality includes:

[0043] Initialize an auxiliary hash table H of size h' and a queued sequence Q of size 2k;

[0044] The B+ tree can be verified by traversing the tree using the KNN search algorithm, determining whether the search sphere centered on the query vector with a preset query radius intersects with the data sphere of the currently traversed node:

[0045] like This indicates that the search sphere intersects with the data sphere of the currently traversed node, and determines whether the currently traversed node is a leaf node:

[0046] If it is not a leaf node, continue traversing its child nodes until a leaf node is reached;

[0047] If it is a leaf node, check the <identifier, index key, high-dimensional vector> of all high-dimensional vectors contained in that leaf node, and determine whether the identifier of the high-dimensional vector is in the auxiliary hash table:

[0048] If the identifier of the high-dimensional vector is in the auxiliary hash table, then the preset distance value in the auxiliary hash table is obtained as the retrieval distance of the high-dimensional vector;

[0049] If the identifier of the high-dimensional vector is not in the auxiliary hash table, the distance between the query vector and the high-dimensional vector is calculated as the retrieval distance of the high-dimensional vector, and <identifier, high-dimensional vector, retrieval distance> is added to the auxiliary hash table based on the identifier and hash value;

[0050] Select a high-dimensional vector whose retrieval distance is less than the preset query radius, and add <identifier, high-dimensional vector, retrieval distance> to the queued sequence;

[0051] like This indicates that the search sphere does not intersect with the data sphere of the currently traversed node, and the search continues to the next node;

[0052] Until the total number of high-dimensional vectors in the queued sequence is not less than the preset number, the identifiers of all high-dimensional vectors in the queued sequence are obtained.

[0053] Based on the identifiers of all high-dimensional vectors in the queued sequence, obtain the corresponding multimodal data, store it in the form of <identifier, multimodal data, high-dimensional vector>, and construct the query result set R.

[0054] Preferably, based on the query result set, the center point, radius, and summary of the nodes, and the root node signature value, a set of verification objects is generated for the query result set, including:

[0055] The initialization verification object collection is empty;

[0056] The KNN search algorithm is used to traverse the verifiable B+ tree and determine whether the search sphere centered on the query vector with a preset verification radius intersects with the data sphere of the currently traversed node.

[0057] like This indicates that the search sphere intersects with the data sphere of the currently traversed node, and it determines whether the currently traversed node is a leaf node.

[0058] If it is a leaf node, then check if the identifier of that leaf node exists in the query result set:

[0059] If it exists, add the center point and radius of the node, as well as the <identifier, index key, and high-dimensional vector> of all high-dimensional vectors in the node, to the verification object set;

[0060] If it does not exist, add the summary of the node, along with the <identifier, high-dimensional vector> of all high-dimensional vectors in the node, to the set of verification objects;

[0061] If it is not a leaf node, then the center point, radius, and the <index key> of all high-dimensional vectors in the node are added to the verification object set;

[0062] like If the search sphere does not intersect with the data sphere of the currently traversed node, then the center point, radius, and summary of that node are added to the verification object set.

[0063] Add the root node signature value and the digital signature values ​​corresponding to all high-dimensional vectors in all nodes of the query result set to the verification object set, and obtain the verification object set VO of the query object set.

[0064] Preferably, the query user obtains a set of verified query results to obtain the actual retrieval results of the query vector, including:

[0065] Determine if all nodes in the result query set are valid objects:

[0066] If for all id α ∈R and id β For any ∈VO-R, there exists a dist(q,V) that does not satisfy... α ) <dist(q,V βIf the node in the result query set is not a valid object, then the current retrieval fails.

[0067] If for all id α ∈R and id β ∈VO-R, all satisfy dist(q,V) α ) <dist(q,V β If the result query set contains valid objects, then all nodes in the result query set are valid objects. Based on the information contained in the valid object set, the minimum subtree is reconstructed, and the summary of all nodes in the minimum subtree is calculated to reconstruct the summary of the root node and obtain the reconstructed root summary.

[0068] Use the public key to verify whether the reconstructed root digest matches the root node signature value:

[0069] If there is no match, the validation of the object set fails, and the current retrieval fails.

[0070] If a match is found, the public key is used to verify the high-dimensional vector and number in each node of the minimum subtree.

[0071] Are the signature values ​​consistent?

[0072] If there is no match, the current search fails;

[0073] If they match, the query result set is true, and the retrieval is successful.

[0074] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0075] The secure and verifiable cross-modal retrieval method described in this invention, based on a multimodal dataset, extracts high-dimensional vectors corresponding to the image and text modalities of all multimodal data. Then, it uses clustering to generate a unified index structure for each node, constructing corresponding image-verifiable B+ trees and text-verifiable B+ trees. This allows for retrieval on the corresponding type of verifiable B+ tree based on the query request type of the user, achieving cross-modal retrieval. The verifiable B+ tree constructed in this invention has a unified index structure for leaf nodes, internal nodes, and the root node, effectively organizing multimodal data and mapping high-dimensional data to a one-dimensional space. This allows for quick location of regions that may contain the target data during queries, reducing the search space and improving query efficiency. Simultaneously, by calculating the center point, radius, and summary of each node, it facilitates rapid determination of the distance between the node and the query vector during subsequent queries, obtaining subtrees that meet the conditions, avoiding unnecessary subtree traversal, and improving query efficiency. Furthermore, this application performs a signature operation on the summary of the root node of the verifiable B+ tree. The query user can verify the signature to determine whether the data has been tampered with, thereby ensuring the authenticity of the query results.

[0076] This invention, after a user submits a query vector, utilizes the KNN search algorithm to traverse a verifiable B+ tree. Centered on the query vector, a search sphere with a pre-defined verification radius is used to obtain vectors that meet the distance requirements, forming a query result set. This avoids meaningless over-searching, saves computational resources and time costs, and improves retrieval efficiency. Simultaneously, this invention strategically utilizes shared tree nodes and prunes unnecessary subtrees, employing different processing methods for intersecting and non-intersecting cases. It comprehensively records relevant information of related nodes, generating a corresponding set of verification objects for the query result set, enabling rapid and effective determination of the authenticity of the query results.

[0077] When verifying the authenticity of the query result set, this invention checks whether all valid objects are in the query result set, reconstructs the minimum subtree based on the verification object set, and verifies whether the reconstructed root digest of the minimum subtree matches the root node signature. It also uses a public key to verify whether the high-dimensional vector in the node of the minimum subtree is consistent with the digital signature value, thereby determining whether the query results in the query result set are accurate and have not been tampered with, further ensuring the integrity and correctness of the query results in the entire cross-modal retrieval process. Attached Figure Description

[0078] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0079] Figure 1 This is a flowchart of the secure and verifiable cross-modal retrieval method provided by the present invention;

[0080] Figure 2 It is a system architecture diagram;

[0081] Figure 3 This is a flowchart demonstrating the cross-modal retrieval results of VCMR, using image and text as examples.

[0082] Figure 4 This is a diagram of a multimodal neural network model;

[0083] Figure 5 This is a schematic diagram illustrating the construction process of a verifiable B+ tree;

[0084] Figure 6 This is a diagram illustrating the generation of the verification object set. Detailed Implementation

[0085] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0086] Reference Figure 1The flowchart shown illustrates the steps of the secure and verifiable cross-modal retrieval method provided by this invention. The specific steps include:

[0087] S101: The data owner extracts features from the text and image modalities of the multimodal data in the multimodal dataset, obtains the high-dimensional vectors of the corresponding modalities, and constructs a verifiable B+ tree for the corresponding modality, including:

[0088] Cluster high-dimensional vectors, use the clusters as leaf nodes, and construct the internal nodes and root node of the B+ tree based on the range of index key values ​​of all high-dimensional vectors in all leaf nodes to form the basic B+ tree;

[0089] Calculate the center point and radius of each node based on its membership relationship;

[0090] Using a hash function, a hash chain is generated in a bottom-up manner, and the digest of each node is calculated;

[0091] Store the node's center point, radius, summary, and the index key values ​​of all high-dimensional vectors.

[0092] Store it in the basic B+ tree as a verifiable B+ tree;

[0093] S102: Digitally sign the root node of the verifiable B+ tree for each modality and obtain the corresponding root node signature value.

[0094] S103: Encrypt each multimodal data in the multimodal dataset to obtain encrypted multimodal data; digitally sign the high-dimensional vector of the corresponding mode for each encrypted multimodal data to obtain the digital signature value of the corresponding mode;

[0095] S104: Upload the verifiable B+ tree, root node signature value, all encrypted multimodal data, and digital signature value to the data service provider for storage;

[0096] S105: The data service provider obtains the query vectors generated by the querying user based on different modalities; based on the modality of the query vector, traverses the verifiable B+ tree corresponding to another modality:

[0097] If the current traversal point is an internal node, then traverse its child nodes;

[0098] If the current traversal point is a leaf node, calculate the distance between the center point of the leaf node and the query vector, obtain the k vectors with the smallest distance, and construct the query result set; based on the query result set, the center point, radius, summary of the node, and the signature value of the root node, generate a set of verification objects for the query result set so that the query user can obtain the verified query result set and obtain the real retrieval result of the query vector.

[0099] In step S101, the high-dimensional vectors corresponding to the text modality and image modality in each multimodal data are obtained, including:

[0100] S101-1: Input the image data from the multimodal data into the image modality deep neural network to obtain the high-dimensional image vector corresponding to the multimodal data;

[0101] S101-2: Use the bag-of-words model to convert text data in multimodal data into bag-of-words vectors, and input the bag-of-words vectors into a text modality deep neural network to obtain high-dimensional text vectors corresponding to multimodal data.

[0102] The image modality deep neural network is a convolutional neural network, comprising, in series: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer. The text modality deep neural network comprises, in series: a first fully connected layer with ReLU activation and a second fully connected layer with identity activation.

[0103] Specifically, nodes in a B+ tree include leaf nodes, internal nodes, and a root node. Based on the membership relationships of each node, the center point and radius of each node are calculated, including:

[0104] Calculate the center point of the leaf node based on all the high-dimensional vectors in the cluster represented by the leaf node. Represented as: m represents the total number of high-dimensional vectors in the cluster represented by the leaf node L, V i Let L represent the i-th high-dimensional vector in the cluster represented by the leaf node L, where 1 ≤ i ≤ m;

[0105] The radius r of a leaf node is obtained based on the distance between the center point of the leaf node and all high-dimensional vectors in the cluster represented by the leaf node. L , is represented as: S represents the set of all high-dimensional vectors in the clusters represented by the leaf nodes. Represents the center point of the leaf node The distance between the i-th high-dimensional vector in the cluster represented by the leaf node;

[0106] Calculate the geometric center of the center points of all child nodes of an internal node, and use that center point as the center point of the internal node. Represented as: k represents the total number of child nodes of the internal node N. Let represent the center point of the j-th child node of the internal node, where 1 ≤ j ≤ k;

[0107] Calculate the minimum hypersphere radius that can cover all child nodes of the internal node, and use it as the radius r of that internal node. N , is represented as: Indicates the center point of the internal node With the j-th child node N of the internal node j The distance between them Represents the radius of the j-th child node of an internal node;

[0108] Calculate the geometric center of the center point of all child nodes of the root node, and use that center point as the center point of the root node.

[0109] Calculate the minimum hypersphere radius that can cover all child nodes of the root node, and use it as the radius r of the root node. root .

[0110] Specifically, a hash chain is generated in a bottom-up manner using a hash function, and a digest of each node is calculated, including:

[0111] Calculate the summary d of leaf node L L , is represented as:

[0112]

[0113] Calculate the summary d of internal node N N , is represented as:

[0114]

[0115] Calculate the root node's summary d root , is represented as:

[0116]

[0117] Where h() represents a hash function, (id L |K L |V i ) represents the <identifier, index key value\high-dimensional vector> of a leaf node; K1 to K j Let represent the index key values ​​of the 1st to jth child nodes of the internal node N, respectively. to Let K′1 and K′j represent the summaries of the 1st to jth child nodes of the internal node N, respectively. n These represent the index key values ​​of the 1st to nth child nodes of the root node, respectively. to These represent the summaries of the 1st to nth child nodes of the root node, where n represents the total number of child nodes of the root node.

[0118] Specifically, calculating the index key value of a high-dimensional vector includes:

[0119] Based on the cluster center point O in the i-th cluster i Dist(O) to the high-dimensional vector p i ,p), calculate the index key K of the high-dimensional vector. p , is represented as:

[0120] K p =i*ρ+dist(O i ,p);

[0121] Where ρ is the preset partitioning parameter.

[0122] The secure and verifiable cross-modal retrieval method described in this invention, when storing data, extracts high-dimensional vectors corresponding to the image and text modalities of all multimodal data based on a multimodal dataset. Then, it uses clustering to generate a unified index structure for each node, constructing corresponding image-verifiable B+ trees and text-verifiable B+ trees. This allows for retrieval on the corresponding type of verifiable B+ tree based on the query request type of the user, achieving cross-modal retrieval. The verifiable B+ tree constructed in this invention has a unified index structure for leaf nodes, internal nodes, and the root node, effectively organizing multimodal data and mapping high-dimensional data to a one-dimensional space. This allows for quick location of regions that may contain the target data during queries, reducing the search space and improving query efficiency. Simultaneously, by calculating the center point, radius, and summary of each node, it facilitates rapid determination of the distance between the node and the query vector during subsequent queries, obtaining subtrees that meet the conditions, avoiding unnecessary subtree traversal, and further improving query efficiency. Furthermore, this application performs a signature operation on the digest of the root node of the verifiable B+ tree. Querying users can verify the signature to determine whether the data has been tampered with, thereby ensuring the authenticity of the query results.

[0123] In this embodiment of the invention, the query vector generated based on the text modality needs to be searched on the image verifiable B+ tree, and the query vector generated based on the image modality needs to be searched on the text verifiable B+ tree.

[0124] Specifically, in step S105, the step of obtaining the query result set includes:

[0125] S105-1: Initialize an auxiliary hash table H of size h' and a queued sequence Q of size 2k;

[0126] S105-2: By traversing the B+ tree using the KNN search algorithm, we can determine whether the search sphere centered on the query vector with a preset query radius intersects with the data sphere of the currently traversed node.

[0127] S105-3: If This indicates that the search sphere intersects with the data sphere of the currently traversed node, and determines whether the currently traversed node is a leaf node:

[0128] If it is not a leaf node, continue traversing its child nodes until a leaf node is reached;

[0129] If it is a leaf node, check the <identifier, index key, high-dimensional vector> of all high-dimensional vectors contained in that leaf node, and determine whether the identifier of the high-dimensional vector is in the auxiliary hash table:

[0130] If the identifier of the high-dimensional vector is in the auxiliary hash table, then the preset distance value in the auxiliary hash table is obtained as the retrieval distance of the high-dimensional vector;

[0131] If the identifier of the high-dimensional vector is not in the auxiliary hash table, the distance between the query vector and the high-dimensional vector is calculated as the retrieval distance of the high-dimensional vector, and <identifier, high-dimensional vector, retrieval distance> is added to the auxiliary hash table based on the identifier and hash value;

[0132] Select a high-dimensional vector whose retrieval distance is less than the preset query radius, and add <identifier, high-dimensional vector, retrieval distance> to the queued sequence;

[0133] S105-4: If This indicates that the search sphere does not intersect with the data sphere of the currently traversed node, and the search continues to the next node;

[0134] S105-5: Until the total number of high-dimensional vectors in the queued sequence is not less than the preset number, obtain the identifiers of all high-dimensional vectors in the queued sequence;

[0135] S105-6: Based on the identifiers of all high-dimensional vectors in the queued sequence, obtain the corresponding multimodal data, store it in the form of <identifier, multimodal data, high-dimensional vector>, and construct the query result set R.

[0136] The data service provider generates a set of verification objects for the query result set based on the query result set, the center point, radius, summary of the nodes, and the root node signature value, including:

[0137] The initialization verification object collection is empty;

[0138] The KNN search algorithm is used to traverse the verifiable B+ tree and determine whether the search sphere centered on the query vector with a preset verification radius intersects with the data sphere of the currently traversed node.

[0139] like This indicates that the search sphere intersects with the data sphere of the currently traversed node, and it determines whether the currently traversed node is a leaf node.

[0140] If it is a leaf node, then check if the identifier of that leaf node exists in the query result set:

[0141] If it exists, add the center point and radius of the node, as well as the <identifier, index key, and high-dimensional vector> of all high-dimensional vectors in the node, to the verification object set;

[0142] If it does not exist, then the summary of that node, and the < of all high-dimensional vectors in that node.

[0143] Identifier, high-dimensional vector, add to the verification object set;

[0144] If it is not a leaf node, then the center point, radius, and the <index key> of all high-dimensional vectors in the node are added to the verification object set;

[0145] like If the search sphere does not intersect with the data sphere of the currently traversed node, then the center point, radius, and summary of that node are added to the verification object set.

[0146] Add the root node signature value and the digital signature values ​​corresponding to all high-dimensional vectors in all nodes of the query result set to the verification object set, and obtain the verification object set VO of the query object set.

[0147] The query retrieves a set of verified query results from the user, yielding the actual retrieval results for the query vector, including:

[0148] Determine if all nodes in the result query set are valid objects:

[0149] If for all id α ∈R and id β For any ∈VO-r, there exists a dist(q,V) that does not satisfy... α ) <dist(q,V β If the node in the result query set is not a valid object, then the current retrieval fails.

[0150] If for all id α ∈R and id β ∈VO-R, all satisfy dist(q,V) α ) <dist(q,V β If the result query set contains valid objects, then all nodes in the result query set are valid objects. Based on the information contained in the valid object set, the minimum subtree is reconstructed, and the summary of all nodes in the minimum subtree is calculated to reconstruct the summary of the root node and obtain the reconstructed root summary.

[0151] Use the public key to verify whether the reconstructed root digest matches the root node signature value:

[0152] If there is no match, the validation of the object set fails, and the current retrieval fails.

[0153] If a match is found, the public key is used to verify whether the high-dimensional vector in each node of the minimum subtree is consistent with the digital signature value.

[0154] If there is no match, the current search fails;

[0155] If they match, the query result set is true, and the retrieval is successful.

[0156] This invention, after a user submits a query vector, utilizes the KNN search algorithm to traverse a verifiable B+ tree. Centered on the query vector, a search sphere with a pre-defined verification radius is used to obtain vectors that meet the distance requirements, forming a query result set. This avoids meaningless over-searching, saves computational resources and time, and improves retrieval efficiency. Simultaneously, this invention strategically utilizes shared tree nodes and prunes unnecessary subtrees, employing different processing methods for intersecting and disjoint cases. It comprehensively records relevant information of related nodes, generating a corresponding verification object set for the query result set, enabling rapid and effective determination of the query result's authenticity. When verifying the authenticity of the query result set, this invention checks whether all valid objects are in the query result set, reconstructs a minimum subtree based on the verification object set, and verifies whether the reconstructed root digest of the minimum subtree matches the root node signature. It uses a public key to verify whether the high-dimensional vectors in the nodes of the minimum subtree are consistent with the digital signature value, thereby determining whether the query results in the query result set are accurate and have not been tampered with, further ensuring the reliability of the query results and the authenticity of the data in the query results throughout the cross-modal retrieval process.

[0157] Based on the above embodiments, the verifiable cross-modal retrieval method VCMR provided by this invention consists of five key algorithms, namely Π = (KeyGen, Construct, Token, Query, Verify); the specific framework includes:

[0158] ① Key generation (KeyGen);

[0159] {<sk,pk> , <K pri ,K pub >}←KenGen(λ);

[0160] The data owner (DO) uses the security parameter λ as input to generate a private key sk and a public key pk for the Paillier cryptosystem, as well as a private key K for digital signatures. pri and public key K pub ;

[0161] ② Index construction (Construct);

[0162] I←Construct(V,pk);

[0163] DO obtains the image and text vectors of the multimodal dataset D, encrypts and signs the dataset, and then applies k-means clustering to construct VB. + The tree structure is then uploaded to the data service provider (DSP), along with the encrypted dataset, signature, and tree structure.

[0164] ③ Generate a token by querying the token;

[0165] q←Token(q,pk);

[0166] The query user QU generates a query vector q and submits it to the data service provider DSP.

[0167] ④ Query processing;

[0168] {R,VO}←Query(q,I);

[0169] Data service provider DSP collaborates with data assistance provider DAP to search for the k most relevant results based on query vectors and verifiable indexes by securely measuring similarity, and returns an encrypted result R and a verification object VO;

[0170] ⑤ Verify the results;

[0171] {0,1}←Verify(R,VO);

[0172] The query user QU uses the information in the verification object VO to verify the integrity and correctness of the encrypted result R.

[0173] The index structure of VCMR is as follows: VB + The tree is used to guarantee the authenticity of the results, but does not provide privacy protection; its nodes contain pointers p. j Index key K j The center point of the cluster and radius and abstract Information such as leaf nodes. <id j ,K j V j Yes, id j It is V j The identity is associated with the original data, the digest is generated through a specific hash calculation, the node digest is calculated from bottom to top, and finally the root node digest is signed for result verification.

[0174] It can be verified that a B+ tree (VB+ tree) can be formalized as follows:

[0175]

[0176] This invention uses the k-means clustering method to partition a high-dimensional space and selects reference points according to specific rules; it maps all vectors to a one-dimensional space and then calculates the center point and radius of each node; based on this, a bottom-up approach is used to determine the summary of each node, and finally, the summary of the root node is signed.

[0177] The cross-modal retrieval method of this invention includes data preprocessing by the data owner, such as generating a key, obtaining a vector, encrypting signature data, and constructing a tree structure for uploading to a data service provider. After the query user submits a query vector, the data service provider and a data assistance provider collaborate to calculate the distance between the query vector and objects in the verifiable B+ tree to obtain a result set. Simultaneously, the data service provider traverses the verifiable B+ tree again to generate verification objects containing information related to the objects in the result set and their signatures. After receiving the result set and verification objects returned by the data service provider, the query user performs a series of operations to ensure the authenticity of the results, including checking whether all valid objects are in the result set, reconstructing and verifying the root digest of the verifiable B+ tree (calculated by the information on the path from the leaf node containing the object to the root node and compared with the signature in the verification object), and confirming the correctness of the object signatures in the result set. This determines whether the retrieval results are accurate and have not been tampered with, ensuring the authenticity of the results throughout the cross-modal retrieval process.

[0178] Based on the above embodiments, this embodiment utilizes the secure and verifiable cross-modal retrieval method provided above for data storage and cross-modal retrieval; refer to Figure 2 The diagram shown is a system architecture diagram; refer to... Figure 3 The diagram shows a flowchart illustrating the verifiable cross-modal retrieval results of VCMR, using image and text as examples. The specific steps include:

[0179] S201: Generate representation vectors using deep neural networks;

[0180] Taking image and text modalities as examples, feature learning is performed using deep learning networks;

[0181] In a two-modal system, a multimodal dataset containing both modalities is represented as:

[0182] n represents the data size;

[0183] DO starts with a multimodal dataset and obtains image vectors for similarity measurement. and text vector

[0184] Reference Figure 4 The diagram shown is a multimodal neural network model; the feature learning part contains two deep neural networks, one for the image modality and the other for the text modality.

[0185] The deep neural network used for image modalities is a convolutional neural network (CNN). This CNN model has 8 layers and outputs the learned image features. Its network structure is as follows:

[0186] 1) conv1: It has 64 convolution kernels, the kernel size is 11×11, the convolution stride is 4×4, the padding is 0, local response normalization (LRN) is applied and 2×2 pooling is performed.

[0187] 2) conv2: 265 convolutional kernels, size 5×5, stride 1×1, padding 2, applying LRN and performing 2×2 pooling;

[0188] 3) conv3: 265 convolutional kernels, size 3×3, stride 1×1, padding 1;

[0189] 4) conv4: 265 convolutional kernels, size 3×3, stride 1×1, padding 1;

[0190] 5) conv5: 265 convolutional kernels, size 3×3, stride 1×1, padding 1, and 2×2 pooling;

[0191] 6) Fully connected layer (full6): has 4096 nodes;

[0192] 7) Full7: Has 4096 nodes;

[0193] 8) Full8: The number of output nodes is the dimension of the high-dimensional vector, that is, the output of this layer is a high-dimensional vector of the image.

[0194] Furthermore, the first seven layers of this network structure all use the Rectified Linear Unit (ReLU) as the activation function, while the eighth layer uses the identity function as the activation function.

[0195] Refer to Table 1 for the configuration parameters of the image modality deep neural network;

[0196] Table 1. Configuration parameters for image modality deep neural network

[0197]

[0198]

[0199] To learn features from the text modality, each text y is first represented as a vector with a Bag of Words (BOW) representation; then, the Bag of Word vector is used as input to a deep neural network with two fully connected layers, denoted as "full1-full2". Referring to Table 2, the parameters for the text deep neural network are configured, where the configuration shows the number of nodes in each layer, the activation function for the first layer is ReLU, and the activation function for the second layer is the identity function.

[0200] Table 2 Configuration parameters for text deep neural networks

[0201] Full1 Number of nodes and output dimensions: 8192 full2 Number of nodes and output dimension: Dimensions of text vectors

[0202] S202: Construct a verifiable B+ tree;

[0203] Generating high-dimensional vectors and Then, the multimodal dataset D is encrypted, and each element is signed.

[0204] For images, by identifying them Raw data sum vector Combine and generate signatures using asymmetric encryption (such as RSA). Then, DO to V I and V T K-means clustering is applied, and a verifiable B+ tree is constructed for each modality using the algorithm I←Construct(V,pk).

[0205] The index implementation vector V of the B+ tree can be verified. i One-dimensional bond K i The indexing steps include:

[0206] ① Clustering: Clustering methods are used to divide points in a high-dimensional space into different regions; the number of partitions (a key hyperparameter) directly affects search performance;

[0207] ② Reference point selection: For each partition P i Select cluster center point O i Used to set the keys for vectors within that partition;

[0208] ③ One-dimensional representation: All vectors are mapped to a one-dimensional space;

[0209] Key K of data point p p The calculation is as follows:

[0210] K p =i*ρ+dist(O i ,p);

[0211] Where i represents the i-th partition, ρ is chosen to be large enough to prevent key overlap between partitions, and dist(O i ,p) is from O i The distance to p is calculated using this key method, which facilitates range queries.

[0212] This invention effectively verifies results by constructing a verifiable B+ tree and utilizing the properties of Merkle trees, preventing data tampering and omissions, and ensuring the authenticity of search results. By constructing a verifiable B+ tree index, multimodal data is effectively organized, mapping high-dimensional data to a one-dimensional space, facilitating rapid location and retrieval of relevant data. During the query process, clustering information and distance calculations are used to reduce unnecessary search scope and improve query speed.

[0213] Reference Figure 5 The diagram illustrates the verifiable construction process of a B+ tree. First, 3-means clustering is applied to partition the space into partitions P1, P2, and P3. Then, a basic B+ tree is constructed based on the K value. The specific construction process includes:

[0214] ① Using the selected clustering method, calculate the centroid of each node. and radius r,

[0215] For example, Use the selected clustering method;

[0216] ② Determine the summary (d) for each node in a bottom-up manner; if it is a leaf node, use... Calculate its summary, if it is an internal node. It is N i The first child node, for example,

[0217] ③ Sign the root node's digest for result verification; that is, s Root =sign(d Root ,K Pri Finally, the encrypted multimodal dataset D, the data signature, and all verifiable B+ trees are uploaded to the DSP.

[0218] S203: Query processing procedure;

[0219] Upon receiving a query, where the query user QU generates a query vector q using the algorithm Token, the DSP calculates the Euclidean distance between the query and objects in the verifiable B+ tree to measure similarity. After receiving the query vector, the DSP runs {R,VO}←Query(q,I) to output the result R and the verification object VO.

[0220] The specific process of obtaining results and verifying objects includes:

[0221] ① Input the query vector q and the search parameter k;

[0222] ② Using the KNN search algorithm in VCMR, with the query vector q, the root node Root of the verifiable B+ tree, and the search parameter k as inputs, output the result queue R, and the initial value is set as

[0223] Before the search, initialize an auxiliary hash table H and a sorted queue Q, whose sizes are |H|=n and |Q|=2k respectively. H is used to prevent redundant dist(q,V i ) calculation, while Q is used to store objects that meet the search criteria. The search radius r will be continuously expanded until the termination condition is met, and matching objects are added to Q.

[0224] During the traversal of the verifiable B+ tree, there may be two situations:

[0225] Intersection: If dist(q,φ node )<r+r node , then the search sphere centered at intersects with the data sphere contained in the node, which indicates that there may be matching items. For leaf nodes, we check all <id i ,K i ,V i > pairs. If id i is in H, we obtain dist from H i ; otherwise, calculate dist i =dist(q,V i ), and use the hash value H(id i ) to insert <id i ,V i ,dist i > into H. If dist i <r, add <id i ,V i ,dist i > to Q. For non-leaf nodes, we continue to traverse their child nodes.

[0226] Disjoint: If dist(q,φ node )≥r+r node , the two spheres do not intersect, and none of the objects in the node meet the search conditions, so no further search is required. After fully expanding the search radius, once |Q|≥R, the KNN search ends. Retrieve semantically relevant multimodal data using the collected identity information Retrieve semantically relevant multimodal data Therefore, the result set R is constructed as It contains identifiers, multimodal data, and their corresponding vectors.

[0227] S204: Verification object generation;

[0228] After traversing the verifiable B+ tree to obtain the retrieval result (R) using the KNN search algorithm in the VCMR method, the DSP traverses the verifiable B+ tree again to generate VOs for all objects in R, strategically utilizing shared tree nodes and pruning unnecessary subtrees to optimize the process.

[0229] This process uses the VO generation algorithm from the VCMR method. bas The VO generation process is explained here using a verifiable B+ tree as input:

[0230] After the kNN search, a shared validation object VO is generated for the result set R. The VO is generated initially as an empty object, taking the query vector q, the result queue R, and the root node of the verifiable B+ tree as input.

[0231] The initial search radius is set to r. R V k It is the k-th vector in R.

[0232] There are two main cases when traversing a verifiable B+ tree:

[0233] Intersection situation:

[0234] For leaf nodes, if the node contains the id from R... i Then r node Add to VO; if the id in R does not exist in the node. i Then d node Add to VO.

[0235] For internal nodes r node Add it to the VO and continue exploring its child nodes.

[0236] Non-intersecting cases Will r node ,d node Add to VO. Afterwards, DSP will sign the root node. Root and the signature of an element in R Added to VO.

[0237] Reference Figure 6The diagram shown illustrates the generation of the verification object set; assuming there is a text vector q, k=2, the result set... DSP according to Calculate r. While traversing the verifiable B+ tree, nodes N1 and N2 are found to intersect with the search radius, so their child nodes are further examined. Node N3 does not satisfy the intersection condition, so its center point is... radius and summary Add to VO. For leaf node L1, since... and Will Included in VO. For leaf nodes L2 and L4, because... and Therefore, the corresponding tuples are also included in the VO. For leaf node L3, since Will Add to VO. Finally, add s Root and Add to VO. The summaries for leaf nodes and internal nodes are calculated as follows:

[0238]

[0239] S205: Verification of the authenticity of the results;

[0240] Upon receiving R and VO, QU uses the auxiliary proof information provided in VO to run {0,1}←Verify(R,VO) to verify the integrity and correctness of R. The specific process is as follows:

[0241] 1) Ensure that all valid objects are in R, that is, for all and All Established;

[0242] 2) Rebuild and validate VB + Tree root summary d Root Specifically, QU first reconstructs a minimal subtree based on the information contained in the VO, which contains all nodes relevant to the query. Then, QU reconstructs the root digest d by computing summaries of these nodes. Root and using public key K pub Verify s Root Does it match the reconstructed root summary?

[0243] 3) Confirm Signature The correctness of QU. QU uses public key K. pub Verify each signature i Does it match the corresponding data object and vector? Figure 6 Taking the example in the example, QU first checks for all and Does it meet the requirements? Then, rebuild VB. + Find the smallest subtree of the tree and verify the conditions of the pruned branches. Using this subtree, QU can reconstruct d Root Using public key K pub Verify s Root Finally, verify the signature. The accuracy of the results is ensured through these steps.

[0244] The verifiable cross-modal retrieval method described in this invention, based on a multimodal dataset, extracts high-dimensional vectors corresponding to the image and text modalities of all multimodal data. Then, it uses clustering to generate a unified index structure for each node, constructing corresponding image-verifiable B+ trees and text-verifiable B+ trees. This allows for retrieval on the corresponding type of verifiable B+ tree based on the query request type of the user, achieving cross-modal retrieval. The verifiable B+ tree constructed in this invention has a unified index structure for leaf nodes, internal nodes, and the root node, effectively organizing multimodal data and mapping high-dimensional data to a one-dimensional space. This allows for quick location of regions that may contain the target data during queries, reducing the search space and improving query efficiency. Simultaneously, by calculating the center point, radius, and summary of each node, it facilitates rapid determination of the distance between the node and the query vector during subsequent queries, obtaining subtrees that meet the conditions, avoiding unnecessary subtree traversal, and improving query efficiency. Furthermore, this application performs a signature operation on the summary of the root node of the verifiable B+ tree. The query user can verify the signature to determine whether the data has been tampered with, thereby ensuring the authenticity of the query results. This invention, after a user submits a query vector, utilizes the KNN search algorithm to traverse a verifiable B+ tree. Centered on the query vector, a search sphere with a pre-defined verification radius is used to obtain vectors that meet the distance requirements, forming a query result set. This avoids meaningless over-searching, saves computational resources and time, and improves retrieval efficiency. Simultaneously, this invention strategically utilizes shared tree nodes and prunes unnecessary subtrees, employing different processing methods for intersecting and disjoint cases. It comprehensively records relevant information of related nodes, generating a corresponding verification object set for the query result set, enabling rapid and effective determination of the query result's correctness and completeness. When verifying the authenticity of the query result set, this invention checks whether all valid objects are in the query result set, reconstructs a minimum subtree based on the verification object set, and verifies whether the reconstructed root digest of the minimum subtree matches the root node signature. It uses a public key to verify whether the high-dimensional vectors in the nodes of the minimum subtree are consistent with the digital signature value, thereby determining whether the query results in the query result set are accurate and have not been tampered with, further ensuring the integrity and correctness of the query results throughout the cross-modal retrieval process.

[0245] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0246] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0247] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0248] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0249] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A secure and verifiable cross-modal retrieval method, characterized in that, include: The data owner extracts features from the text and image modalities of the multimodal data in the multimodal dataset, obtains high-dimensional vectors for the corresponding modalities, and constructs verifiable B+ trees for the corresponding modalities, including: Cluster high-dimensional vectors, use the clusters as leaf nodes, and construct the internal nodes and root node of the B+ tree based on the range of index key values ​​of all high-dimensional vectors in all leaf nodes to form the basic B+ tree; Based on the membership relationships of each node, calculate the center point and radius of each node, including: calculating the center point of the leaf node based on all high-dimensional vectors in the cluster represented by the leaf node. , is represented as: ; Leaf nodes The total number of high-dimensional vectors in the represented cluster. Leaf nodes The first cluster represented by the cluster is the A high-dimensional vector, The radius of a leaf node is obtained based on the distance between its center point and all high-dimensional vectors in the cluster it represents. , is represented as: ; This represents the set of all high-dimensional vectors in the cluster represented by the leaf node. Represents the center point of the leaf node The leaf node represents the first cluster of clusters. The distance between each high-dimensional vector; calculate the geometric center of the center point of all child nodes of an internal node, and use this center point as the center point of that internal node. , is represented as: ; Indicates internal nodes The total number of child nodes, The first internal node represents the... The center point of each child node Calculate the minimum hypersphere radius that can cover all child nodes of the internal node, and use this radius as the radius of the internal node. , is represented as: ; Indicates the center point of the internal node With the internal node's first child nodes The distance between them The first internal node represents the... The radius of each child node; calculate the geometric center of the center point of all child nodes of the root node, and use this center point as the center point of the root node. Calculate the minimum hypersphere radius that can cover all child nodes of the root node, and use this radius as the radius of the root node. ; Using a hash function, a hash chain is generated in a bottom-up manner, and the digest of each node is calculated; The center point, radius, summary, and index key values ​​of all high-dimensional vectors of the nodes are stored in the base B+ tree as a verifiable B+ tree. Digitally sign the root node of the verifiable B+ tree for each modality and obtain the corresponding root node signature value; Encrypt each multimodal data point in the multimodal dataset to obtain encrypted multimodal data; digitally sign the high-dimensional vector corresponding to each encrypted multimodal data point to obtain the digital signature value of the corresponding modality; The verifiable B+ tree, root node signature value, all encrypted multimodal data, and digital signature value will be uploaded to the data service provider for storage. The data service provider obtains query vectors generated by users based on different modalities; based on the modality of the query vector, it traverses the verifiable B+ tree corresponding to another modality: If the current traversal point is an internal node, then traverse its child nodes; If the current traversal point is a leaf node, calculate the distance between the center point of the leaf node and the query vector, obtain the k vectors with the smallest distance, and construct the query result set; based on the query result set, the center point, radius, summary of the node, and the signature value of the root node, generate a set of verification objects for the query result set so that the query user can obtain the verified query result set and obtain the real retrieval result of the query vector.

2. The secure and verifiable cross-modal retrieval method according to claim 1, characterized in that, Using a hash function, a hash chain is generated in a bottom-up manner, and the digest of each node is calculated, including: Calculate leaf nodes Abstract , is represented as: ； Calculate internal nodes Abstract , is represented as: ； Calculate the summary of the root node , is represented as: ； in, Represents a hash function. The <identifier, index key value\high-dimensional vector> represents the leaf node; to Representing internal nodes respectively The first to the first The index key value of each child node. to Representing internal nodes respectively The first to the first A summary of each child node; to These represent the first to the second elements of the root node, respectively. The index key value of each child node. to These represent the first to the second elements of the root node, respectively. A summary of each child node. This represents the total number of child nodes of the root node.

3. The secure and verifiable cross-modal retrieval method according to claim 1, characterized in that, Calculating the index key value of a high-dimensional vector includes: Based on the Cluster centroids in each cluster To high-dimensional vectors distance Calculate the index key value of a high-dimensional vector. , is represented as: ； in, These are the preset partition parameters.

4. The secure and verifiable cross-modal retrieval method according to claim 1, characterized in that, Obtain the high-dimensional vectors corresponding to the text modality and image modality in each multimodal dataset, including: Image data from multimodal data is input into an image modality deep neural network to obtain high-dimensional image vectors corresponding to the multimodal data. The bag-of-words model is used to convert text data in multimodal data into bag-of-words vectors, and the bag-of-words vectors are input into a text modality deep neural network to obtain high-dimensional text vectors corresponding to the multimodal data.

5. The secure and verifiable cross-modal retrieval method according to claim 4, characterized in that, The image modality deep neural network is a convolutional neural network, comprising the following layers connected in series: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer.

6. The secure and verifiable cross-modal retrieval method according to claim 4, characterized in that, The text modality deep neural network includes, in sequence, a first fully connected layer with ReLU activation function and a second fully connected layer with identity function activation function.

7. The secure and verifiable cross-modal retrieval method according to claim 1, characterized in that, Based on the query vector modality, traverse the verifiable B+ tree corresponding to another modality, including: Initialize a size of auxiliary hash table With a size of Queued sequence ; The B+ tree can be verified by traversing the tree using the KNN search algorithm, determining whether the search sphere centered on the query vector with a preset query radius intersects with the data sphere of the currently traversed node: like If the search sphere intersects with the data sphere of the currently traversed node, then determine whether the currently traversed node is a leaf node: If it is not a leaf node, continue traversing its child nodes until a leaf node is reached; If it is a leaf node, check the <identifier, index key, high-dimensional vector> of all high-dimensional vectors contained in that leaf node, and determine whether the identifier of the high-dimensional vector is in the auxiliary hash table: If the identifier of the high-dimensional vector is in the auxiliary hash table, then the preset distance value in the auxiliary hash table is obtained as the retrieval distance of the high-dimensional vector; If the identifier of the high-dimensional vector is not in the auxiliary hash table, the distance between the query vector and the high-dimensional vector is calculated as the retrieval distance of the high-dimensional vector, and <identifier, high-dimensional vector, retrieval distance> is added to the auxiliary hash table based on the identifier and hash value; Select a high-dimensional vector whose retrieval distance is less than the preset query radius, and add <identifier, high-dimensional vector, retrieval distance> to the queued sequence; like If the search sphere does not intersect with the data sphere of the currently traversed node, then continue searching for the next node. Until the total number of high-dimensional vectors in the queued sequence is not less than the preset number, the identifiers of all high-dimensional vectors in the queued sequence are obtained. Based on the identifiers of all high-dimensional vectors in the queued sequence, the corresponding multimodal data is obtained and stored in the form of <identifier, multimodal data, high-dimensional vector>, thus constructing a query result set. .

8. The secure and verifiable cross-modal retrieval method according to claim 7, characterized in that, Based on the query result set, the node's center point, radius, digest, and root node signature value, a set of verification objects is generated for the query result set, including: The initialization verification object collection is empty; The KNN search algorithm is used to traverse the verifiable B+ tree and determine whether the search sphere centered on the query vector with a preset verification radius intersects with the data sphere of the currently traversed node. like This indicates that the search sphere intersects with the data sphere of the currently traversed node, and it determines whether the currently traversed node is a leaf node. If it is a leaf node, then check if the identifier of the leaf node exists in the query result set: If it exists, add the center point and radius of the node, as well as the <identifier, index key, and high-dimensional vector> of all high-dimensional vectors in the node, to the verification object set; If it does not exist, add the summary of the node, along with the <identifier, high-dimensional vector> of all high-dimensional vectors in the node, to the set of verification objects; If it is not a leaf node, then the center point, radius, and the <index key> of all high-dimensional vectors in the node are added to the verification object set; like If the search sphere does not intersect with the data sphere of the currently traversed node, then the center point, radius, and summary of that node are added to the verification object set. Add the root node signature value and the digital signature values ​​corresponding to all high-dimensional vectors in all nodes of the query result set to the verification object set, and obtain the verification object set of the query object set. .

9. The secure and verifiable cross-modal retrieval method according to claim 8, characterized in that, The query retrieves a set of verified query results from the user, yielding the actual retrieval results for the query vector, including: Determine if all nodes in the result query set are valid objects: If for all and There are dissatisfactions If the node is not a valid object, then the current retrieval will fail if not all nodes in the result query set are valid objects. If for all and All satisfy If all nodes in the result query set are valid objects, then the minimum subtree is reconstructed based on the information contained in the verification object set, and the summary of all nodes in the minimum subtree is calculated to reconstruct the summary of the root node and obtain the reconstructed root summary. Use the public key to verify whether the reconstructed root digest matches the root node signature value: If there is no match, the validation of the object set fails, and the current retrieval fails. If a match is found, the public key is used to verify whether the high-dimensional vector in each node of the minimum subtree is consistent with the digital signature value. If there is no match, the current search fails; If they match, the query result set is true, and the retrieval is successful.

Images