A semantic-aware cross-modal encrypted retrieval method

Abstract

This invention relates to a semantically aware cross-modal encrypted retrieval method, belonging to the fields of information retrieval and data encryption. By introducing deep learning, it enhances the semantic feature mining capability for multimodal data, replacing the traditional keyword retrieval mode with semantic feature retrieval. A low-overhead multimodal data encrypted retrieval method is introduced, designing a parallel retrieval tree structure based on the block-based approach to achieve low-overhead, privacy-preserving, and fast semantic similarity retrieval. The rationality of the scheme is analyzed from aspects such as precision, search time, and storage overhead. This invention obtains multimodal features by extracting features from query requests; it uses a similarity-hiding encrypted retrieval algorithm to generate encrypted query trapdoors for multimodal query features; and it enables secure multimodal data retrieval in the medical IoT, combining the powerful edge computing capabilities and fast response of traditional scenarios, overcoming the problems of insufficient end-user computing resources and key leakage.

Description

Technical Field

[0001] This invention belongs to the field of information retrieval and data encryption, and relates to a semantically aware cross-modal encrypted retrieval method for application in the Internet of Things for Medical Use. Background Technology

[0002] With the advancement of big data and artificial intelligence technologies, the scale of the Internet of Things (IoT) in healthcare is continuously expanding. Healthcare IoT data includes patient personal information, medical records, diagnostic results, treatment plans, and genomic data, providing valuable data resources for medical research, public health policy formulation, drug development, and precision medicine, and possessing immense commercial value. However, healthcare IoT data, such as patient identity information and diagnostic conclusions, involves patient privacy. As healthcare IoT infrastructure migrates fully to cloud server platforms, healthcare IoT cloud platforms face significant privacy risks. Traditional encryption methods typically require data decryption before search matching, which not only increases the risk of data leakage but also requires substantial computing resources for encryption and decryption, significantly impacting the real-time nature of search result feedback. Currently, searchable encryption technology has been applied to encrypted retrieval of text data in healthcare IoT. However, healthcare IoT data is multimodal; for example, text data such as patient descriptions and pathological features in medical documents may contain medical images such as MRI and CT scans. Traditional searchable encryption technology is no longer suitable for complex multimodal healthcare IoT scenarios. Therefore, how to achieve multimodal querying while protecting the privacy of healthcare IoT data has become a new challenge in the utilization of healthcare IoT data.

[0003] Currently, there are several cross-modal encrypted retrieval schemes. Compared with unimodal retrieval, multimodal retrieval avoids the drawbacks of relying on only one modality. It can provide additional information, forming the advantage of information complementarity, thereby improving the accuracy of the model. For example, GUO.C et al. proposed in “Enabling secure cross-modal retrieval over encrypted heterogeneous IoT databases with collective matrix factorization” [in IEEE Internet of Things Journal, 2020, 7(4): 3104-3113] to achieve secure cross-modal retrieval of encrypted heterogeneous IoT databases through collective matrix factorization. This scheme combines collective matrix factorization and homomorphic encryption, and uses locality-sensitive hashing to build a privacy-preserving index structure, realizing text-to-image and image-to-text retrieval. However, due to the use of homomorphic encryption and collective matrix factorization, it brings disastrous computational overhead and low retrieval speed. Zhou et al., in "Privacy-Preserving Image Retrieval with Multi-Modal Query" [in The Computer Journal, 2023: bxad117], designed a hybrid image retrieval framework that integrates visual and textual features at the decision level to construct an inverted index and implement multi-keyword image queries based on a Bloom filter. While this model optimizes the retrieval structure, it ignores the catastrophic increase in index length caused by data diversity. Furthermore, the aforementioned study separates the textual and image modalities, failing to analyze the semantic relevance features of multimodal text and image data. The lack of integration and utilization of semantic features from multimodal data during the search process limits search performance in both areas.

[0004] While past research on cross-modal encrypted retrieval has yielded some results, multimodal encrypted retrieval in the context of the Internet of Things (IoT) for healthcare still faces many challenges. On the one hand, the demands of the IoT for healthcare necessitate high real-time performance and accuracy. On the other hand, it is also limited by the semantic understanding required for complex queries. Summary of the Invention

[0005] In view of this, the technical problem to be solved by the present invention is that: existing searchable encryption methods for medical IoT cannot meet the needs of multimodal retrieval; searchable encryption schemes ignore the semantic information association between different modalities of the same meta-entity, resulting in insufficient cross-modal retrieval accuracy; feature extractors designed manually based on expert experience cannot express the complex information of multimodal data; and existing encryption retrieval methods that leak similarity are difficult to achieve low-overhead, privacy-preserving, and fast semantic similarity retrieval.

[0006] The purpose of this invention is to provide a semantically aware cross-modal encrypted retrieval method.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A semantically aware cross-modal encrypted retrieval method includes the following steps:

[0009] S1: Use a pre-trained semantic-aware multimodal feature extraction method to extract features from the query request and obtain multimodal features;

[0010] S2: Using a similarity-hidden encrypted retrieval algorithm, generate encrypted query trapdoors for multimodal query features and upload them to the cloud;

[0011] S3: Cross-modal data encryption retrieval: Utilizing a parallel balanced binary tree index structure, similarity retrieval of encrypted domains is performed in the cloud, and similarity is ranked by semantic relevance measurement before returning the retrieval results.

[0012] Furthermore, S1 specifically includes the following steps:

[0013] S11: Model pre-training: The image encoder is initialized using the ResNet50 model, and the text encoder is initialized using PubMed BERT parameters;

[0014] S12: Add a non-linear mapping head to map the feature vectors generated by the encoder to a uniform length, and use a non-linear activation function to enable the network to learn complex feature representations;

[0015] S13: Optimize the loss function by maximizing the inner product of image-text pairs;

[0016] S14: Construct N positive samples from N image-text pairs, N 2 -N negative samples are used for training, where positive samples are the corresponding image-text pair feature vectors and negative samples are non-corresponding image-text pair feature vectors;

[0017] S15: Use cosine similarity (sim) to measure the correspondence between image-text pairs. The larger the sim value, the stronger the correspondence.

[0018] S16: The training task is to optimize the objective, expressed by the following formula:

[0019]

[0020] Among them, I i Let T be the feature vector of the i-th image. j Let j be the feature vector of the j-th text.

[0021] S17: The loss function is the cross-entropy comparison loss function, expressed as the following formula:

[0022]

[0023] in, For image-to-text contrast loss, τ represents the text-to-image contrast loss, N is the number of samples in the batch, and τ is the temperature parameter.

[0024] S18: Assume that the feature vector of the multimodal data image is f i =(f i,1 ,f i,2 ,...,f i,l The corresponding text feature vector is t. i =(t i,1 ,t i,2 ,...,t i,l Both are l-dimensional; the features of the corresponding image and text are concatenated into a 2l-dimensional feature vector, which satisfies the following formula:

[0025] f fusion_i =(t i ,f i )=(t i,1 ,t i,2 ,...,t i,l ,f i,1 ,f i,2 ,...,f i,l )

[0026] Among them, f fusion_i Fusion features that preserve multimodal semantic details;

[0027] S19: Query image vector q i =(q i,1 ,q i,2 ,...,q i,l ) and text feature vector q t =(q t,1 ,p t,2 ,...,p t,l When generating a query trapdoor, the edge server concatenates the text and image feature vectors according to the query request; for a single-modal query, alignment is achieved by padding with zeros.

[0028] Furthermore, S2 specifically includes the following steps:

[0029] S21: Generate a key sk = (S, M1, M2) based on the security parameter λ, where M1 and M2 are d-dimensional random invertible square matrices, and S is a d-dimensional random 0 / 1 sequence; assume P is the set of vectors to be compared, and q is the query vector with dimension d;

[0030] S22: Noise addition, for vector p in set P i Normalization is performed, noise is added to the feature vector, and it is expanded to (d+2) dimensions; where the (d+1)th dimension is a random number δ, and the (d+2)th dimension is 1;

[0031] S23: Normalize the query vector q and then expand it to (d+1) dimensions, with the (d+1)th dimension being 1. Perturb this total (d+1)-dimensional vector using a random number β (0 < β < 1), resulting in {βq, β}. Then add a random number c to the (d+2)-dimensional vector. Finally:

[0032] S24: According to the splitting rules of the random sequence S, the vector is split.

[0033] Where, if the i-th bit of S is 0 (i.e., S[i] = 0), then take in It is a random number; when the i-th bit is 1 (i.e., S[i] = 0), in, and It is a random number.

[0034] S25: The dot product method is used to compare the similarity of query q in set P, expressed as the following formula:

[0035] Score(Enc(p i ),Enc(q))=β(p i ·q+δ)+c

[0036] Among them, Enc(p i ) is the index of the ciphertext state, Enc(q) is the query of the ciphertext state, (p i ·q) represents the inner product similarity between the medical multimodal data feature vector and the query feature vector, where δ is a random number and c is a random number;

[0037] S26: Preserve the encrypted function relationship for queries with the same keyword; through noise perturbation, utilize the randomness of δ and c to convert the similarity measure into a size relationship;

[0038] S27: During the query, the similarity between the index vectors p1 and p2 of different medical multimodal data in the encrypted state and the query q is used to sort and retrieve the data.

[0039] Furthermore, S3 specifically includes the following steps:

[0040] S31: Key generation. Calculate m = d / n as the number of blocks. Given the security parameter λ, generate the symmetric encryption key κ and the indexed encryption key SK, expressed as the following formula:

[0041] SK = (S, M1, M2)

[0042] Where S = {S1, S2, ..., S} m}, (S j ∈ R {0,1} d ), M1={M 1,1 ,...,M 1,j ,...,M 1,m}, M2={M 2,1 ,...,M 2,j ,...,M 2,m}, M 1,j and M 2,j Let m be a d-dimensional random invertible matrix, and m ≠ d;

[0043] S32: Index construction. Based on the cluster center vectors after clustering, the data is divided into m blocks, each with a length of n. Each cluster is represented by C blocks. i ={c1,...,c j ,...,c m For the j-th block, using secure asymmetric scalar-product-preserving encryption (sASPE), we obtain the following formula:

[0044] SK j ={S j M 1,j M 2,j}

[0045] Encrypt it using the following formula:

[0046]

[0047] The encrypted dimension is 2d, and the encrypted index vector of this cluster is obtained:

[0048]

[0049] S33: Each node in the index tree is represented as:

[0050] node=<nID,V,left,right,CID / FID>

[0051] Where nID is the identifier of the node, left and right are pointers to the left and right child nodes, CID is the medical classification cluster identifier corresponding to the node, FID is the medical IoT multimodal data identifier corresponding to the node, and FID only exists in leaf nodes and is 0 in non-leaf nodes; V is used to store specific features. In leaf nodes, V stores medical multimodal data, while in non-leaf nodes it represents the matching rules from bottom to top.

[0052] S34: Data encryption, using the symmetric encryption key κ to encrypt document F i Plaintext data is encrypted into ciphertext data E. i ; Encrypt index I F ={I i} and ciphertext data E={E i Upload the data to the cloud server; retrieve the data based on the clustering center; construct a forest structure retrieval system to encrypt the data and build a corresponding index.

[0053] S35: Data Update. After receiving a data update request from the data owner, the cloud server first calculates the cluster with the highest similarity among all clusters, and then enters the tree of that cluster to construct a node, represented as:

[0054] node=<nID,I i ,left,right,FID>

[0055] Insert the tree to update the index. In the clustered search tree, the leaf nodes represent the file identifier rather than the cluster identifier.

[0056] S36: The data user, acting as a hospital staff member, transmits the query to the edge server. The edge server divides the query vector into m blocks of length n, q = {q1, q2, ..., q...} j ,...,q m For the j-th block, the ciphertext domain similarity-hidden encryption algorithm is used with SK. j ={S j M 1,j M 2,j Encrypt it as Finally, an encrypted query is obtained.

[0057] S37: Upon receiving a query request from the edge server Then, using a block-parallel retrieval method, parallel retrieval is performed to obtain the most matching cluster. Finally, the top k most matching documents E are retrieved from the subtrees within that cluster. kAt this point, the length of the algorithm vector is 2d, so the number of blocks is m, and the block length is 2n; and the V of each tree node is a 2d-dimensional vector.

[0058] S38: The edge server receives the encrypted data set E returned by the cloud server. k Then, the data is decrypted using the decryption key to obtain plaintext medical multimodal data, which is then transmitted to the data user.

[0059] Furthermore, S24 also includes the following steps:

[0060] After splitting the vector, it is encrypted using random invertible matrices M1 and M2, as shown in the following formula:

[0061]

[0062] Furthermore, the sorting retrieval described in S27 is expressed as the following formula:

[0063] Score(Enc(p2),Enc(q))-Score(Enc(p1)Enc(q))=β(p2·q-p1·q)

[0064] Here, β is only related to the query, and different queries have different β values. Therefore, this improved scheme can resist backdoor attacks and prevent the leakage of original similarity results between different queries.

[0065] Furthermore, S37 specifically includes:

[0066] The vectors of the query trapdoor and the root node of the index tree are divided into m blocks, and multiple subqueries are processed in parallel in different subtrees.

[0067] Initialize a stack S, push the root node onto the stack. If the element is a leaf node and its relevance score to the query vector is higher than the lowest score kthscore in the current Top-k results, then update the list of the top k most relevant documents, insert the document ID into it, and remove the document with the lowest score.

[0068] For non-leaf nodes, calculate the relevance scores of their left and right child nodes to the query trapdoor; decide whether to push the child node onto the stack and continue traversing based on the relevance scores; if the relevance score is higher than kthscore, push the child node with higher relevance onto the stack, otherwise continue processing the top element of the stack.

[0069] After completing the traversal of the index tree, select the top-k relevant documents from the merged results, and perform a final score check and update to ensure that the returned documents are the most relevant.

[0070] Return a list of the top k most relevant documents.

[0071] The beneficial effects of this invention are as follows:

[0072] By introducing deep learning, the semantic feature mining capability for multimodal data is enhanced, replacing the traditional keyword retrieval mode with semantic feature retrieval. A low-overhead multimodal data encryption retrieval method is introduced, and a parallel retrieval tree structure is designed based on the block idea to achieve fast semantic similarity retrieval with low overhead and privacy protection. The rationality of the solution is analyzed from the aspects of precision, search time, and storage overhead. Secure retrieval of multimodal data in the medical Internet of Things is realized, which combines the powerful edge computing capabilities and fast response characteristics, and overcomes the problems of insufficient end-user computing resources and key leakage in traditional scenarios.

[0073] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0074] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0075] Figure 1 This is a diagram of a semantically aware medical IoT multimodal data security semantic search architecture in an embodiment of the present invention;

[0076] Figure 2 This is a schematic diagram of the semantically aware multimodal data feature extraction method in an embodiment of the present invention;

[0077] Figure 3 This is a schematic diagram of the low-overhead multimodal data encryption retrieval method in an embodiment of the present invention. Detailed Implementation

[0078] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0079] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0080] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0081] Please see Figure 1 This is a semantically aware medical IoT multimodal data security semantic search architecture diagram in an embodiment of the present invention. This architecture consists of a three-layer architecture: edge-cloud, integrating the advantages of edge servers being close to the querying user for real-time query processing, powerful computing capabilities for data processing, privacy protection, and non-distribution of keys. Specifically, the architecture comprises three layers: the terminal usage layer, the edge computing layer, and the cloud retrieval layer.

[0082] The terminal layer is mainly responsible for sensing and collecting retrieval needs. Considering the privacy of medical data, medical staff will pass the generated unimodal or multimodal query requests to the edge server.

[0083] The edge (edge ​​server) computing layer mainly includes semantic-aware feature extraction, trapdoor generation, and encryption. Semantic-aware feature extraction obtains single or multiple modal data from query requests. Semantic features are extracted using semantic-aware multimodal data feature extraction methods to construct plaintext query trapdoors. The plaintext trapdoors are then encrypted using low-overhead multimodal data encryption retrieval methods to generate the final trapdoor, which is then transmitted to the cloud server.

[0084] The 5G / 6G core network servers in the cloud retrieval layer are primarily responsible for storing encrypted multimodal medical IoT data and retrieving encrypted query trapdoors from the edge to perform the final similarity ranking and return the results. Here, a parallel balanced binary tree hierarchical index structure is used for retrieval, and the top-k results with privacy-preserving similarity are returned to the edge server for decryption, and finally delivered to medical workers to assist in medical diagnosis.

[0085] This invention proposes a semantically aware cross-modal encrypted retrieval method, which mainly includes the following steps:

[0086] S1: Use a pre-trained semantic-aware multimodal feature extraction method to extract features from the query request and obtain multimodal features.

[0087] S2: Using a similarity-hidden encrypted retrieval algorithm, generate encrypted query trapdoors for multimodal query features and upload them to the cloud.

[0088] S3: Cross-modal data encryption retrieval: Utilizing a parallel balanced binary tree index structure, similarity retrieval of encrypted domains is performed in the cloud, and similarity is ranked by semantic relevance measurement before returning the retrieval results.

[0089] Please see Figure 2 This is a schematic diagram of the semantic-aware multimodal data feature extraction method in an embodiment of the present invention. S1 specifically includes the following steps:

[0090] S11: Model pre-training: After verification, the image encoder was initialized using the classic ResNet50 model, and the text encoder was initialized using PubMedBERT parameters.

[0091] S12: Adding a non-linear mapping head maps the feature vectors generated by the encoder to a uniform length. The non-linear activation function allows the network to learn more complex feature representations.

[0092] S13: The loss function is optimized by maximizing the inner product of the image-text pair, so that the image feature extractor and text feature extractor for the medical field can extract features better after training.

[0093] S14: Construct N positive samples from N image-text pairs, N 2 -N negative samples are used for training, where positive samples are the corresponding image-text pair feature vectors, and negative samples are non-corresponding image-text pair feature vectors.

[0094] S15: Use cosine similarity to measure the correspondence between image-text pairs. The larger the cosine similarity sim, the stronger the correspondence, meaning that the two different modalities are more likely to describe the same thing.

[0095] S16: The training task can be represented as an optimization objective, expressed by the following formula:

[0096]

[0097] Among them, I i Let T be the feature vector of the i-th image. jLet j be the feature vector of the j-th text.

[0098] S17: The loss function is the cross-entropy comparison loss function, expressed as the following formula:

[0099]

[0100] in, For image-to-text contrast loss, τ represents the text-to-image contrast loss, N is the number of samples in the batch, and τ is the temperature parameter.

[0101] S18: Assume we have multimodal data image feature vector f i =(f i,1 ,f i,2 ,...,f i,l ) and the corresponding text feature vector t i =(t i,1 ,t i,2 ,...,t i,l Each feature vector has a length of l. The features of the corresponding image and text are concatenated into a 2l-dimensional feature vector, ensuring that the detailed semantic features of both the image and text are preserved, i.e., satisfying the following formula:

[0102] f fusion_i =(t i ,f i )=(t i,1 ,t i,2 ,...,t i,l ,f i,1 ,f i,2 ,...,f i,l )

[0103] Among them, f fusion_i Fusion features that preserve multimodal semantic details.

[0104] S19: Query image vector q i =(q i,1 ,q i,2 ,...,q i,l ) and text feature vector q t =(q t,1 ,p t,2 ,...,p t,l When generating query trapdoors, the edge server concatenates text and image feature vectors according to the query request, and for unimodal queries, it aligns them by padding with zeros.

[0105] Please see Figure 3This is a schematic diagram of the low-overhead multimodal data encryption retrieval method in this embodiment of the invention; the features obtained from S1 need to be encrypted to prevent leakage. Therefore, a similarity-hiding encryption method is used to generate an encrypted query trapdoor for the multimodal query features and upload it to the cloud. S2 specifically includes the following steps:

[0106] S21: Generate a key sk = (S, M1, M2) based on the security parameter λ, where M1 and M2 are d-dimensional random invertible square matrices, and S is a d-dimensional random 0 / 1 sequence. Assume P is the set of vectors to be compared, and q is the query vector with dimension d.

[0107] S22: Noise addition, for vector p in set P i First, normalization is performed. For safety reasons, noise is added to the feature vector, expanding it to (d+2) dimensions. Here, the (d+1)th dimension is a random number δ, and the (d+2)th dimension is 1.

[0108] S23: For the query vector q, first normalize it, then expand it to (d+1) dimensions, with the (d+1)th dimension being 1. Then, perturb this total (d+1)-dimensional vector using a random number β (0 < β < 1), resulting in {βq, β}. Finally, add a random number c to the (d+2)-dimensional vector, resulting in the final vector.

[0109] S24: Split the vector according to the random sequence S.

[0110] Where, if the i-th bit of S is 0 (i.e., S[i] = 0), then take in It is a random number; when the i-th bit is 1 (i.e., S[i] = 0), in It is a random number It is a random number; after splitting, it is encrypted according to the random invertible matrices M1 and M2:

[0111]

[0112] S25: To compare the similarity of query q in set P while ensuring safety, the dot product method is used: Score(Enc(p) i ),Enc(q))=β(p i ·q+δ)+c, where (p i ·q) represents the inner product similarity between the feature vectors of the medical multimodal data and the query feature vector, i.e., the correlation between the document and the query, where Enc(p) i ) is the index of the ciphertext state, Enc(q) is the query of the ciphertext state, (p i·q) represents the inner product similarity between the medical multimodal data feature vector and the query feature vector, δ is a random number, and c is a random number.

[0113] S26: Under the original scheme, (p i •q) This would reveal the specific numerical value of similarity. The proposed method preserves the encrypted functional relationship of queries with the same keyword. By using noise perturbation and leveraging the randomness of δ and c, the similarity measurement is reflected through a magnitude relationship, rather than a precise numerical relationship, thus avoiding the problem of similarity leakage.

[0114] S27: During a query, the data is sorted and retrieved by measuring the similarity between the index vectors p1 and p2 of different medical multimodal data in the encrypted state and the query q.

[0115] Score(Enc(p2),Enc(q))-Score(Enc(p1)Enc(q))=β(p2·q-p1·q)

[0116] Here, β is only related to the query, and different queries have different β values. Therefore, this improved scheme can resist backdoor attacks and prevent the leakage of original similarity results between different queries.

[0117] S3: Utilizing a parallel balanced binary tree index structure, similarity retrieval of the encrypted domain is performed in the cloud. Similarity is ranked based on semantic relevance and the retrieval results are returned. The cross-modal data encryption retrieval process specifically includes the following steps:

[0118] S31: Key generation. Calculate m = d / n as the number of blocks. Given the security parameter λ, generate the symmetric encryption key κ and the indexed encryption key SK, expressed as the following formula:

[0119] SK = (S, M1, M2)

[0120] Where S = {S1, S2, ..., S} m}, (S j ∈ R {0,1} d ), M1={M 1,1 ,...,M 1,j ,...,M 1,m}, M2={M 2,1 ,...,M 2,j ,...,M 2,m}, M 1,j and M 2,j Let m be a d-dimensional random invertible matrix, and m ≠ d.

[0121] S32: Index construction. Based on the cluster center vectors after clustering, the data is divided into m blocks, each with a length of n. Each cluster is represented by C blocks.i ={c1,...,c j ,...,c m For the j-th block, using secure asymmetric scalar-product-preserving encryption (sASPE), we obtain the following formula:

[0122] SK j ={S j M 1,j M 2,j}

[0123] Encrypt it using the following formula:

[0124]

[0125] The encrypted dimension is 2d, and the encrypted index vector of this cluster is obtained:

[0126]

[0127] S33: Each node in the index tree is represented as:

[0128] node=<nID,V,left,right,CID / FID>

[0129] Where nID is the identifier of the node, left and right are pointers to the left and right child nodes, CID is the medical classification cluster identifier corresponding to the node, and FID is the medical IoT multimodal data identifier corresponding to the node. FID only exists in leaf nodes and is 0 in non-leaf nodes. V stores specific features. In leaf nodes, V stores medical multimodal data, while in non-leaf nodes, it represents the matching rules from bottom to top.

[0130] S34: Data encryption, using the symmetric encryption key κ to encrypt document F i Plaintext data is encrypted into ciphertext data E. i Then encrypt the index And the encrypted data E = {E i The data is uploaded to the cloud server. All data is first retrieved based on cluster centers, and then a forest structure is built for retrieval; that is, all data undergoes encryption and indexing.

[0131] S35: Data Update. After receiving a data update request from the data owner, the cloud server first calculates the cluster with the highest similarity among all clusters, and then enters the tree of that cluster to construct a node, represented as:

[0132] node=<nID,I i,left,right,FID>

[0133] Insert the tree to update the index. In the clustered search tree, the leaf nodes represent the file identifier rather than the cluster identifier.

[0134] S36: The data user, acting as a hospital staff member, transmits the query to the edge server. The edge server divides the query vector into m blocks of length n, q = {q1, q2, ..., q...} j ,...,q m For the j-th block, the ciphertext domain similarity-hidden encryption algorithm is used with SK. j ={S j M 1,j M 2,j Encrypt it as Finally, an encrypted query is obtained.

[0135] S37: Upon receiving a query request from the edge server Then, using a block-parallel retrieval method, parallel retrieval is performed to obtain the most matching cluster. Finally, the top k most matching documents E are retrieved from the subtrees within that cluster. k Because of the encryption algorithm, the length of the algorithm vector is 2d, so the number of blocks is m, and the block length is 2n. Furthermore, the V of each tree node is a 2d-dimensional vector.

[0136] Specifically, the vectors of the query trapdoor and the root node of the index tree are divided into m blocks, and processed in parallel across different subtrees using multiple subqueries. First, a stack S is initialized, and the root node is pushed onto the stack. If the element is a leaf node and its relevance score to the query vector is higher than the lowest score (kthscore) in the current Top-k results, the list of the top k most relevant documents is updated, the document ID is inserted, and the document with the lowest score is removed. For non-leaf nodes, the relevance scores of their left and right child nodes to the query trapdoor are calculated. The decision to push the child node onto the stack and continue traversal is based on the relevance score. If the relevance score is higher than kthscore, the child node with higher relevance is pushed onto the stack; otherwise, the top element of the stack is processed. After completing the traversal of the index tree, the Top-k relevant documents are selected from the merged results, and a final score check and update are performed to ensure that the returned documents are indeed the most relevant. Finally, the algorithm returns a list of the top k most relevant documents.

[0137] S38: The edge server receives the encrypted data set E returned by the cloud server. k Then, the data is decrypted using the decryption key to obtain plaintext medical multimodal data, which is then transmitted to the data user.

[0138] In summary, this paper proposes a semantic feature fusion-based multimodal encrypted retrieval algorithm for medical IoT, based on searchable encryption technology and a multimodal semantic feature extraction model, to achieve encrypted retrieval of multimodal data in medical IoT. This invention proposes a semantically aware multimodal data feature extraction method. Existing methods, based on traditional keyword extraction techniques, struggle to perform semantic abstraction on text and image multimodal data. The proposed method enhances the semantic feature mining capability for multimodal data by designing a semantic feature extraction module for medical multimodal data, replacing the traditional keyword retrieval mode with semantic feature retrieval.

[0139] This invention proposes a low-overhead, multimodal data encryption retrieval method. Existing encryption retrieval methods have significant search and storage overhead. The proposed method improves upon the existing ASPE encryption algorithm, protects the privacy information of ciphertext domain similarity, enhances the level of privacy protection, and designs a parallel retrieval tree structure based on the block-based approach, achieving low-overhead, privacy-preserving, and fast semantic similarity retrieval.

[0140] This invention also proposes a secure semantic search architecture for multimodal data in the medical IoT. Traditional encrypted retrieval architectures are designed for single-modal data in the medical IoT and cannot meet the search needs of multimodal data. The proposed architecture is based on the edge-cloud collaboration concept, integrating the advantages of multimodal technology and searchable encryption technology. The edge is responsible for generating trapdoors and decrypting multimodal data, while the cloud is responsible for storing multimodal data and semantic retrieval in encrypted space. For the first time, this architecture enables users to perform low-latency, high-precision retrieval of multimodal data while ensuring the privacy of medical IoT users.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A semantically aware cross-modal encrypted retrieval method, characterized in that, Includes the following steps: S1: Use a pre-trained semantic-aware multimodal feature extraction method to extract features from the query request and obtain multimodal features; S2: Using a similarity-hidden encrypted retrieval algorithm, generate encrypted query trapdoors for multimodal query features and upload them to the cloud; S3: Cross-modal data encryption retrieval: Utilizing a parallel balanced binary tree index structure, similarity retrieval of encrypted domains is performed in the cloud, and similarity is ranked by semantic relevance measurement and the retrieval results are returned; S1 specifically includes the following steps: S11: Model pre-training: The image encoder is initialized using the ResNet50 model, and the text encoder is initialized using PubMedBERT parameters; S12: Add a non-linear mapping head to map the feature vectors generated by the encoder to a uniform length, and use a non-linear activation function to enable the network to learn complex feature representations; S13: Optimize the loss function by maximizing the inner product of image-text pairs; S14: Construct N positive samples from N image-text pairs, N 2 -N negative samples are used for training, where positive samples are the corresponding image-text pair feature vectors and negative samples are non-corresponding image-text pair feature vectors; S15: Use cosine similarity (sim) to measure the correspondence between image-text pairs. The larger the sim value, the stronger the correspondence. S16: The training task is to optimize the objective, expressed by the following formula: Among them, I i Let T be the feature vector of the i-th image. j Let j be the feature vector of the j-th text. S17: The loss function is the cross-entropy comparison loss function, expressed as the following formula: in, For image-to-text contrast loss, τ represents the text-to-image contrast loss, N is the number of samples in the batch, and τ is the temperature parameter. S18: Assume that the feature vector of the multimodal data image is f i =(f i,1 ,f i,2 ,...,f i,l The corresponding text feature vector is t. i =(t i,1 ,t i,2 ,...,t i,l Both have a length of l; the features of the corresponding image and text are concatenated into a 2l-dimensional feature vector, which satisfies the following formula: f fusion_i =(t i ,f i )=(t i,1 ,t i,2 ,...,t i,l ,f i,1 ,f i,2 ,...,f i,l ) Among them, f fusion_i Fusion features that preserve multimodal semantic details; S19: Query image vector q i =(q i,1 ,q i,2 ,...,q i,l ) and text feature vector q t =(q t,1 ,p t,2 ,...,p t,l When generating a query trapdoor, the edge server concatenates the text and image feature vectors according to the query request; for single-modal queries, alignment is achieved by padding with zeros. S2 specifically includes the following steps: S21: Generate a key sk = (S, M1, M2) based on the security parameter λ, where M1 and M2 are d-dimensional random invertible matrices, and S is a d-dimensional random 0 / 1 sequence; assume P is the set of vectors to be compared, and q is the query vector with dimension d; S22: Noise addition, for vector p in set P i Normalization is performed, noise is added to the feature vector, and it is expanded to d+2 dimensions; where the (d+1)th dimension is a random number δ, and the (d+2)th dimension is 1; S23: Normalize the query vector q, then expand it to d+1 dimensions, with the d+1th dimension being 1; perturb this total d+1-dimensional vector using random numbers β, 0 < β < 1, resulting in {βq, β}; then add random numbers c to the d+2-dimensional vector, finally: S24: According to the splitting rules of the random sequence S, the vector is split. Where, if the i-th bit of S is 0, that is, S[i] = 0, then take in It is a random number; when the i-th bit is 1, that is, S[i] = 1, in, and It is a random number. S25: The dot product method is used to compare the similarity of query q in set P, expressed as the following formula: Score(Enc(p i ),Enc(q))=β(p i ·q+δ)+c Among them, Enc(p i ) represents the index of the ciphertext state, Enc(q) represents the query of the ciphertext state, and p i ·q represents the inner product similarity between the medical multimodal data feature vector and the query feature vector, δ is a random number, and c is a random number; S26: Preserve the encrypted function relationship for queries with the same keyword; through noise perturbation, utilize the randomness of δ and c to convert the similarity measure into a size relationship; S27: During the query, the similarity between the index vectors p1 and p2 of different medical multimodal data in the encrypted state and the query q is used to sort and retrieve the data. S3 specifically includes the following steps: S31: Key generation. Calculate m = d / n as the number of blocks. Given the security parameter λ, generate the symmetric encryption key κ and the indexed encryption key SK, expressed as the following formula: SK = (S, M1, M2) Where S = {S1, S2, ..., S} m }, (S j ∈ R {0,1} d ), M1={M 1,1 ,...,M 1,j ,...,M 1,m }, M2={M 2,1 ,...,M 2,j ,...,M 2,m }, M 1,j and M 2,j Let m be a d-dimensional random invertible matrix, and m ≠ d; S32: Index construction. Based on the cluster center vectors after clustering, the data is divided into m blocks, each with a length of n. Each cluster is represented by C blocks. i ={c1,...,c j ,...,c m For the j-th block, using secure asymmetric scalar product encryption, we obtain the following formula: EN j ={S j ,M 1,j ,M 2,j } Encrypt it using the following formula: The encrypted dimension is 2d, and the encrypted index vector of this cluster is obtained: S33: Each node in the index tree is represented as: node=<nID,V,left,right,CID / FID> Where nID is the identifier of the node, left and right are pointers to the left and right child nodes, CID is the medical classification cluster identifier corresponding to the node, FID is the medical IoT multimodal data identifier corresponding to the node, and FID only exists in leaf nodes and is 0 in non-leaf nodes; V is used to store specific features. In leaf nodes, V stores medical multimodal data, while in non-leaf nodes it represents the matching rules from bottom to top. S34: Data encryption, using the symmetric encryption key κ to encrypt document F i Plaintext data is encrypted into ciphertext data E. i ; Encrypt index I F ={I i } and ciphertext data E={E i Upload the data to the cloud server; retrieve the data based on the clustering center; construct a forest structure retrieval system to encrypt the data and build a corresponding index. S35: Data Update. After receiving a data update request from the data owner, the cloud server first calculates the cluster with the highest similarity among all clusters, and then enters the tree of that cluster to construct a node, represented as: node=<nID,I i ,left,right,FID> Insert the tree to update the index. In the clustered search tree, the leaf nodes represent the identifier of the medical multimodal data file rather than the cluster identifier. S36: The data user, acting as a hospital staff member, transmits the query to the edge server. The edge server divides the query vector into m blocks of length n, q = {q1, q2, ..., q...} j ,...,q m For the j-th block, the ciphertext domain similarity-hidden encryption algorithm is used with SK. j ={S j M 1,j M 2,j Encrypt it as Finally, an encrypted query is obtained. S37: Upon receiving a query request from the edge server Then, using a block-parallel retrieval method, parallel retrieval is performed to obtain the most matching cluster. Finally, the top k most matching documents E are retrieved from the subtrees within that cluster. k At this point, the length of the algorithm vector is 2d, so the number of blocks is m, and the block length is 2n; and the V of each tree node is a 2d-dimensional vector. S38: The edge server receives the encrypted data set E returned by the cloud server. k Then, the data is decrypted using the decryption key to obtain plaintext medical multimodal data, which is then transmitted to the data user. S24 further includes the following steps: After splitting the vector, it is encrypted using random invertible matrices M1 and M2, as shown in the following formula:

2. The semantically aware cross-modal encrypted retrieval method according to claim 1, characterized in that, The sorting retrieval described in S27 is expressed by the following formula: Score(Enc(p2),Enc(q))-Score(Enc(p1)Enc(q))=β(p2·q-p1·q) Here, β is only related to the query, and different queries have different β values. Therefore, this improved scheme can resist backdoor attacks and prevent the leakage of original similarity results between different queries.

3. The semantically aware cross-modal encrypted retrieval method according to claim 1, characterized in that, Specifically, S37 is: The vectors of the query trapdoor and the root node of the index tree are divided into m blocks, and multiple subqueries are processed in parallel in different subtrees. Initialize a stack S, push the root node onto the stack. If the element is a leaf node and its relevance score to the query vector is higher than the lowest score kthscore in the current Top-k results, then update the list of the top k most relevant documents, insert the document ID into it, and remove the document with the lowest score. For non-leaf nodes, calculate the relevance scores of their left and right child nodes to the query trapdoor; decide whether to push the child node onto the stack and continue traversing based on the relevance scores; if the relevance score is higher than kthscore, push the child node with higher relevance onto the stack, otherwise continue processing the top element of the stack. After completing the traversal of the index tree, select the top-k relevant documents from the merged results, and perform a final score check and update to ensure that the returned documents are the most relevant. Return a list of the top k most relevant documents.

Images