Semantic perception cross-modal encryption retrieval method

Abstract

The invention relates to a cross-modal encryption retrieval method based on semantic perception, and belongs to the field of information retrieval and data encryption. Deep learning is introduced, the semantic feature mining capability oriented to multi-modal data is enhanced, and semantic feature retrieval is used for replacing a traditional keyword retrieval mode; a low-overhead multi-modal data encryption retrieval method is introduced, a parallel retrieval tree structure is designed based on a partitioning thought, and low-overhead and privacy-protected semantic similarity rapid retrieval is realized; the reasonability of the scheme is analyzed from the precision ratio, the search time, the storage overhead and the like. The method comprises the following steps: performing feature extraction on a query request to obtain a multi-modal feature; generating an encrypted query trap door for the multi-modal query features by using a similarity hidden encryption retrieval algorithm; according to the cross-modal data encryption retrieval, multi-modal data security retrieval in the medical internet of things is realized, the characteristics of strong edge computing capability and quick response are integrated, and the problems of insufficient computing resources and key leakage of terminal users in a traditional scene are solved.

Description

Technical Field

[0001] The present invention belongs to the field of information retrieval and data encryption, and relates to a semantically-aware cross-modal encryption retrieval method applied in the medical Internet of Things. Background Art

[0002] With the development and progress of big data and artificial intelligence technology, the scale of medical Internet of Things continues to expand. Medical Internet of Things data includes patients' personal information, medical records, diagnosis results, treatment plans, and genomic data, which can provide valuable data resources for medical research, public health policy formulation, drug development, and precision medicine, and contain great commercial value. However, medical Internet of Things data such as patient identity information and diagnosis conclusions involve patient privacy. With the full migration of medical Internet of Things support infrastructure to cloud server platforms, medical Internet of Things cloud platforms face major risks of privacy leakage. Traditional encryption methods usually require data to be decrypted before search and matching, which not only increases the risk of data leakage, but also requires more computing resources to encrypt and decrypt data. Its complicated encryption and decryption links will greatly affect the real-time feedback of search results. At present, searchable encryption technology has been applied in encrypted retrieval of text data in medical Internet of Things. However, data in medical Internet of Things is multimodal. For example, text data such as patient descriptions and pathological characteristics in medical documents contain medical images such as magnetic resonance imaging and CT. Traditional searchable encryption technology is no longer applicable to complex multimodal medical Internet of Things scenarios. Therefore, how to achieve multimodal query on the basis of protecting the privacy data of medical Internet of Things has become a new challenge in the utilization of medical Internet of Things data.

[0003] At present, there are some cross-modal encrypted retrieval schemes. Compared with single modality, multimodality avoids the disadvantage of relying on only one modality. It can provide additional information and form the advantage of information complementarity, thereby improving the accuracy of the model. For example, GUO.C et al. proposed in "Enabling secure cross-modal retrieval over encrypted heterogeneous IoT databases with collective matrix factorization" [in IEEE Internet of Things Journal, 2020, 7 (4): 3104-3113] to achieve secure cross-modal retrieval of encrypted heterogeneous IoT databases through collective matrix factorization. This scheme combines collective matrix factorization with homomorphic encryption, and uses local sensitive hashing to build a privacy-preserving index structure to achieve text-to-image and image-to-text retrieval. However, due to the use of homomorphic encryption and collective matrix factorization, it brings catastrophic computational overhead and low retrieval speed. Zhou.F et al. designed a hybrid image retrieval framework in "Privacy-Preserving Image Retrieval with Multi-Modal Query" [in The Computer Journal, 2023: bxad117], integrating visual features and text features at the decision level, constructing an inverted index, and implementing multi-keyword image queries based on Bloom filters. Although this model optimizes the retrieval structure, it ignores the catastrophic index length caused by data diversification. In addition, the above research separates the text modality from the image modality, does not analyze the semantic correlation characteristics of text and image multimodal data, and does not integrate and utilize the semantic features of multimodal data in the search process, resulting in limited search performance.

[0004] Although relevant research on cross-modal encrypted retrieval has achieved certain results in the past, multimodal encrypted retrieval in the medical Internet of Things scenario still faces many challenges. On the one hand, it is limited by the needs of the medical Internet of Things scenario, which has high requirements for real-time and accuracy. On the other hand, it is also limited by the semantic understanding of complex query requirements. Summary of the invention

[0005] In view of this, the technical problems to be solved by the present invention are: the existing searchable encryption methods of medical Internet of Things cannot meet the needs of multimodal retrieval; the searchable encryption scheme ignores the semantic information association between different modalities of the same meta-entity, resulting in insufficient cross-modal retrieval accuracy; the feature extractor manually designed based on expert experience cannot express the complex information of multimodal data; the existing encryption retrieval methods with similarity leakage make it difficult to achieve low-overhead, privacy-preserving fast retrieval of semantic similarity.

[0006] The purpose of the present invention is to provide a semantically-aware cross-modal encrypted retrieval method.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A semantically-aware cross-modal encrypted retrieval method comprises the following steps:

[0009] S1: Use the pre-trained semantic-aware multimodal feature extraction method to extract features from the query request and obtain multimodal features;

[0010] S2: Using the similarity-hidden encrypted retrieval algorithm, an encrypted query trap is generated for the multimodal query features and uploaded to the cloud;

[0011] S3: Cross-modal data encrypted retrieval: Use a parallel balanced binary tree index structure to perform similarity retrieval of the ciphertext domain in the cloud, sort the similarities using semantic relevance metrics, and return the retrieval results.

[0012] Further, the S1 specifically includes the following steps:

[0013] S11: Model pre-training, the image encoder is initialized with the ResNet50 model, and the text encoder is initialized with PubMed BERT parameters;

[0014] S12: Add a nonlinear mapping head to map the feature vector generated by the encoder to a uniform length, and use a nonlinear activation function to enable the network to learn complex feature representations;

[0015] S13: Optimize the loss function by maximizing the inner product of image-text pairs;

[0016] S14: Construct N positive samples through N image-text pairs, N 2 -N negative samples are used for training, where the positive samples are the corresponding image-text pair feature vectors, and the negative samples are the non-corresponding image-text pair feature vectors;

[0017] S15: Use cosine similarity sim to measure the correspondence between image-text pairs. The larger the sim, the stronger the correspondence.

[0018] S16: The training task is the optimization target, which is expressed as the following formula:

[0019]

[0020] Among them, I i is the feature vector of the i-th image, T j is the feature vector of the jth text;

[0021] S17: The loss function is the cross entropy contrast loss function, expressed as the following formula:

[0022]

[0023] in, is the image-to-text contrast loss, is the contrast loss from text to image, N is the number of samples in the batch, and τ is the temperature parameter;

[0024] S18: Assume that the image feature vector with multimodal data is f i =(f i,1 ,f i,2 ,...,f i,l ), the corresponding text feature vector is t i =(t i,1 ,t i,2 ,...,t i,l ), both of which are l-dimensional in length; the features of the corresponding image text are concatenated into a 2l-dimensional feature vector, which satisfies the following formula:

[0025] f fusion_i =(t i ,f i )=(t i,1 ,t i,2 ,...,t i,l ,f i,1 ,f i,2 ,...,f i,l )

[0026] Among them, f fusion_i Fusion features preserved for multimodal semantic details;

[0027] S19: Query image vector q i =(q i,1 ,q i,2 ,...,q i,l ) and text feature vector q t =(q t,1 ,p t,2 ,...,p t,l ), when generating a query trapdoor, the edge server concatenates the text and image feature vectors according to the query request; when it comes to unimodal queries, they are aligned by zero-padding.

[0028] Further, the S2 specifically includes the following steps:

[0029] S21: Generate a key sk=(S,M1,M2) according to the security parameter λ, where M1 and M2 are d-dimensional random reversible matrices, and S is a d-dimensional random 0 / 1 sequence; assume that P is the set of vectors to be compared, and q is the query vector with a dimension of d;

[0030] S22: Noise addition, for vector p in set P i , normalize it, add noise to the feature vector, and expand it to (d+2) dimensions; where the d+1th dimension is a random number δ, and the d+2th dimension is 1;

[0031] S23: Normalize the query vector q and expand it to (d+1) dimensions, where the (d+1)th dimension is 1. The total (d+1)-dimensional vector is perturbed by a random number β (0<β<1), and the perturbed vector is {βq,β}. Then, a random number c is added to the (d+2) dimension, and finally:

[0032] S24: splitting the vector according to the splitting rule of the random sequence S;

[0033] If the i-th bit of S is 0 (i.e. S[i] = 0), then in is a random number; when the i-th bit is 1 (i.e. S[i] = 0), in, and is a random number,

[0034] S25: Use the dot product method to compare the similarity of query q in set P, which is expressed as the following formula:

[0035] Score(Enc(p i ),Enc(q))=β(p i ·q+δ)+c

[0036] Among them, Enc(p i ) is the index of the ciphertext state, Enc(q) is the query of the ciphertext state, (p i q) is the inner product similarity between the medical multimodal data feature vector and the query feature vector, δ is a random number, and c is a random number;

[0037] S26: retain the encrypted function relationship of the same keyword query; through noise perturbation, use the randomness of δ and c to convert the similarity measurement into a size relationship;

[0038] S27: When querying, sorting and retrieval are performed by measuring the similarity between different medical multimodal data index vectors p1 and p2 in the encrypted state and the query q.

[0039] Further, the S3 specifically includes the following steps:

[0040] S31: Key generation, calculate m=d / n as the number of blocks, given the security parameter λ, generate the symmetric encryption key κ and the index encryption key SK, expressed as the following formula:

[0041] SK=(S,M1,M2)

[0042] Where S = {S1, S2, ..., S m},(S j ∈ R {0,1} d ), M1={M 1,1 ,...,M 1,j ,...,M 1,m},M2={M 2,1 ,...,M 2,j ,...,M 2,m},M 1,j and M 2,j is a d-dimensional random reversible matrix, and m≠d;

[0043] S32: Index construction, according to the cluster center vector after clustering, it is divided into m blocks, each block is n in length, and each cluster representative is divided into C i ={c1,...,c j ,...,c m}, for the jth block, using secure asymmetric scalar-product-preserving encryption (sASPE), we get the following formula:

[0044] SK j ={S j ,M 1,j ,M 2,j}

[0045] Encrypt it into the following formula:

[0046]

[0047] The encrypted dimension is 2d, and the cluster encrypted index vector is obtained:

[0048]

[0049] S33: Each node on the index tree is represented as:

[0050] node=<nID,V,left,right,CID / FID>

[0051] Among them, nID is the identifier of the node Node, left and right are pointers to the left child node and the right child node, CID is the medical classification cluster identifier corresponding to the node, FID is the corresponding medical Internet of Things multimodal data identifier, and FID only exists in leaf nodes and is 0 in non-leaf nodes; V is to save specific features. In leaf nodes, V saves medical multimodal data, and in non-leaf nodes, it represents the bottom-up matching rules;

[0052] S34: Data encryption, using the symmetric encryption key κ to encrypt the document F i The plaintext data is encrypted into ciphertext data E i ; Encrypt index I F = {I i} and ciphertext data E={E i}Upload to the cloud server; retrieve the above data according to the cluster center; construct a forest structure retrieval to encrypt the above data and construct a corresponding index;

[0053] S35: Data update. After receiving the data update request from the data owner, the cloud server first calculates the cluster with the highest similarity among all clusters, and then enters the tree of the cluster to construct a node, which is expressed as:

[0054] node=<nID,I i ,left,right,FID>

[0055] Insert the tree to update the index. In the clustered search tree, the leaf node represents the file identifier rather than the cluster identifier.

[0056] S36: The data user, as a hospital staff, passes the query to the edge server, which divides the query vector into m blocks of n-length sequences, q = {q1, q2, ..., q j ,...,q m}, for the jth block, use the ciphertext domain similarity hiding encryption algorithm with SK j ={S j ,M 1,j ,M 2,j} Encrypt it into Finally, we get the encrypted query

[0057] S37: After receiving the query request sent by the edge server After that, we use the block parallel retrieval method to perform parallel retrieval to obtain the best matching cluster, and then retrieve the top k best matching documents E in the subtree of the cluster. k; At this time, the length of the algorithm vector is 2d, so the number of blocks is m, and the block length is 2n; and V of each tree node is a 2d-dimensional vector;

[0058] S38: The edge server receives the encrypted data set E returned by the cloud server k Afterwards, it is decrypted according to the decryption key to obtain the plaintext medical multimodal data and passed to the data user.

[0059] Furthermore, the S24 further comprises the following steps:

[0060] After the vector is split, it is encrypted according to the random reversible matrices M1 and M2, which is expressed as the following formula:

[0061]

[0062] Further, the ranking search in S27 is expressed as the following formula:

[0063] Score(Enc(p2),Enc(q))-Score(Enc(p1)Enc(q))=β(p2·q-p1·q)

[0064] Among them, β is only related to the query, and different queries have different β values. Therefore, this improved scheme can resist backward attacks and prevent the original similarity results from being leaked between different queries.

[0065] Further, the S37 is specifically:

[0066] Divide the query trapdoor and the vector of the index tree root node into m blocks and process them in parallel in different subtrees through multiple subqueries;

[0067] Initialize a stack S and push the root node into the stack. If the element is a leaf node and its relevance score to the query vector is higher than the lowest score kthscore in the current Top-k results, update the list of the top k most relevant documents, insert the document ID into it, and remove the document with the lowest score.

[0068] For non-leaf nodes, calculate the correlation scores between its left and right child nodes and the query trapdoor; decide whether to push the child node into the stack and continue traversing based on the correlation score; if the correlation score is higher than kthscore, push the child node with high correlation into the stack, otherwise continue to process the top element of the stack;

[0069] After completing the traversal of the index tree, select the top-k relevant documents from the merged results, and perform a final score check and update to ensure that the returned documents are the most relevant;

[0070] Returns a list of the top k most relevant documents.

[0071] The beneficial effects of the present invention are:

[0072] By introducing deep learning, the ability to mine semantic features for multimodal data is enhanced, and semantic feature retrieval replaces the traditional keyword retrieval mode; a low-overhead multimodal data encryption retrieval method is introduced, and a parallel retrieval tree structure is designed based on the block idea to achieve low-overhead, privacy-preserving semantic similarity fast retrieval; the rationality of the scheme is analyzed from the aspects of precision rate, search time, storage overhead, etc.; and secure retrieval of multimodal data in the medical Internet of Things is achieved, which combines the powerful edge computing capabilities and fast response characteristics to overcome the problems of insufficient computing resources and key leakage of end users in traditional scenarios.

[0073] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0075] Figure 1 This is a diagram of a semantically-aware secure semantic search architecture for multimodal data in the medical Internet of Things in an embodiment of the present invention;

[0076] Figure 2 Schematic diagram of a semantic-aware multimodal data feature extraction method in an embodiment of the present invention;

[0077] Figure 3 Schematic diagram of a low-overhead multimodal data encryption retrieval method in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0079] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0080] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0081] See also Figure 1 , is a diagram of the semantic-aware secure semantic search architecture for multimodal data in the medical Internet of Things in an embodiment of the present invention; the architecture consists of a three-layer architecture of end-edge-cloud, integrating the edge server's proximity to the query user to facilitate real-time processing of query requests, powerful computing to facilitate data processing, privacy protection, and key non-decentralization. Specifically, the architecture consists of three layers: terminal use layer, edge computing layer, and cloud retrieval layer:

[0082] The terminal usage layer is mainly responsible for the perception and collection of retrieval needs. Considering the privacy of medical data, medical staff will pass the generated unimodal or multimodal query requests to the edge server.

[0083] The edge (edge ​​server) computing layer mainly includes semantic-aware feature extraction and trapdoor generation and encryption. Semantic-aware feature extraction obtains single or multiple modal data from the query request, and extracts semantic features through semantic-aware multimodal data feature extraction methods to construct plaintext query trapdoors. The plaintext trapdoors are encrypted through low-overhead multimodal data encryption retrieval methods to generate the final trapdoor and pass it to the cloud server.

[0084] The 5G / 6G core network server of the cloud retrieval layer is mainly responsible for storing the encrypted multimodal medical IoT data and retrieving the encrypted query trapdoors from the edge to perform the final similarity sorting and return the results. Here, the retrieval is performed through the parallel balanced binary tree hierarchical index structure, and the top-k results after similarity privacy protection are returned to the edge server for decryption, and finally passed to medical workers to assist in medical diagnosis.

[0085] The present invention proposes a semantically-aware cross-modal encrypted retrieval method, which mainly includes the following steps:

[0086] S1: Use the pre-trained semantic-aware multimodal feature extraction method to extract features from query requests and obtain multimodal features.

[0087] S2: Using the similarity-hidden encrypted retrieval algorithm, an encrypted query trap is generated for the multimodal query features and uploaded to the cloud.

[0088] S3: Cross-modal data encrypted retrieval: Use a parallel balanced binary tree index structure to perform similarity retrieval of the ciphertext domain in the cloud, sort the similarities using semantic relevance metrics, and return the retrieval results.

[0089] See also Figure 2 , is a schematic diagram of a semantic-aware multimodal data feature extraction method in an embodiment of the present invention, where S1 specifically includes the following steps:

[0090] S11: Model pre-training: After verification, the image encoder (Image Encoder) adopts the ResNet50 classic model initialization, and the text encoder (Text Encoder) selects PubMedBERT parameters for initialization.

[0091] S12: Adding a nonlinear mapping head maps the feature vector generated by the encoder to a uniform length. The nonlinear activation function enables the network to learn more complex feature representations.

[0092] S13: The loss function is optimized by maximizing the inner product of the image-text pair, so that the image feature extractor and text feature extractor for the medical field can better extract features after training.

[0093] S14: Construct N positive samples through N image-text pairs, N 2 -N negative samples are used for training, where the positive samples are the corresponding image-text pair feature vectors, and the negative samples are the non-corresponding image-text pair feature vectors.

[0094] S15: Cosine similarity is used to measure the correspondence between image-text pairs. The larger the cosine similarity sim is, the stronger the correspondence is, that is, the two different modal data are more likely to describe the same thing.

[0095] S16: The training task can be expressed as an optimization objective, expressed as the following formula:

[0096]

[0097] Among them, I i is the feature vector of the i-th image, T jis the feature vector of the jth text;

[0098] S17: The loss function is the cross entropy contrast loss function, expressed as the following formula:

[0099]

[0100] in, is the image-to-text contrast loss, is the contrast loss from text to image, N is the number of samples in the batch, and τ is the temperature parameter.

[0101] S18: Assume that we have a multimodal data image feature vector f i =(f i,1 ,f i,2 ,...,f i,l ) and the corresponding text feature vector t i =(t i,1 ,t i,2 ,...,t i,l ), the length is l-dimensional. The features of the corresponding image and text are concatenated into a 2l-dimensional feature vector to ensure that the detailed semantic features of the image and text are preserved, that is, the following formula is satisfied:

[0102] f fusion_i =(t i ,f i )=(t i,1 ,t i,2 ,...,t i,l ,f i,1 ,f i,2 ,...,f i,l )

[0103] Among them, f fusion_i Fusion features preserved for multimodal semantic details.

[0104] S19: Query image vector q i =(q i,1 ,q i,2 ,...,q i,l ) and text feature vector q t =(q t,1 ,p t,2 ,...,p t,l ), when generating the query trapdoor, the edge server concatenates the text and image feature vectors according to the query request, and for unimodal queries, they are aligned through zero-padding operation.

[0105] See also Figure 3, is a schematic diagram of a low-overhead multimodal data encryption retrieval method in an embodiment of the present invention; the features obtained from S1 need to be encrypted to prevent leakage. Therefore, an encryption method of similarity hiding is used to generate an encrypted query trap for the multimodal query features and upload it to the cloud. S2 specifically includes the following steps:

[0106] S21: Generate a key sk=(S,M1,M2) according to the security parameter λ, where M1,M2 are d-dimensional random reversible matrices, and S is a d-dimensional random 0 / 1 sequence. Assume that P is the set of vectors to be compared, q is the query vector, and its dimension is d.

[0107] S22: Noise addition, for vector p in set P i , first normalize it, and for safety reasons, add noise to the feature vector and expand it to (d+2) dimensions. The d+1th dimension is a random number δ, and the d+2th dimension is 1.

[0108] S23: The query vector q is also normalized first, then expanded to (d+1) dimensions, with the (d+1)th dimension being 1. Then, the total (d+1)-dimensional vector is perturbed by a random number β (0<β<1), and the perturbed vector is {βq,β}. Then, a random number c is added to the (d+2)th dimension, and the final vector is

[0109] S24: Split the vector, and the splitting rule follows the random sequence S.

[0110] If the i-th bit of S is 0 (i.e. S[i] = 0), then in is a random number; when the i-th bit is 1 (i.e. S[i] = 0), in Is a random number It is a random number; after the split is completed, it is encrypted according to the random reversible matrix M1, M2:

[0111]

[0112] S25: How to compare the similarity of query q in set P while ensuring security? The dot product method is used. Score(Enc(p i ),Enc(q))=β(p i ·q+δ)+c, where (p i q) represents the inner product similarity between the medical multimodal data feature vector and the query feature vector, that is, the relevance between the document and the query, where Enc(p i ) is the index of the ciphertext state, Enc(q) is the query of the ciphertext state, (p i·q) is the inner product similarity between the medical multimodal data feature vector and the query feature vector, δ is a random number, and c is a random number.

[0113] S26: Under the original scheme, (p i q) will leak the specific value of similarity. The proposed method retains the encrypted functional relationship of the same keyword query. Through noise perturbation, the randomness of δ and c is used to reflect the similarity measurement through the size relationship, without the existence of an exact numerical relationship, thus avoiding the similarity leakage problem.

[0114] S27: When querying, sorting and retrieval are performed by measuring the similarity between different medical multimodal data index vectors p1 and p2 in the encrypted state and the query q:

[0115] Score(Enc(p2),Enc(q))-Score(Enc(p1)Enc(q))=β(p2·q-p1·q)

[0116] Among them, β is only related to the query, and different queries have different β values. Therefore, this improved scheme can resist backward attacks and prevent the original similarity results from being leaked between different queries.

[0117] S3: Use the parallel balanced binary tree index structure to perform similarity retrieval of the ciphertext domain in the cloud, sort the similarities by semantic relevance measurement, and return the retrieval results. The cross-modal data encryption retrieval process specifically includes the following steps:

[0118] S31: Key generation, calculate m=d / n as the number of blocks, given the security parameter λ, generate the symmetric encryption key κ and the index encryption key SK, expressed as the following formula:

[0119] SK=(S,M1,M2)

[0120] Where S = {S1, S2, ..., S m},(S j ∈ R {0,1} d ), M1={M 1,1 ,...,M 1,j ,...,M 1,m},M2={M 2,1 ,...,M 2,j ,...,M 2,m},M 1,j and M 2,j is a d-dimensional random reversible matrix, and m≠d.

[0121] S32: Index construction, according to the cluster center vector after clustering, it is divided into m blocks, each block is n in length, and each cluster representative is divided into Ci ={c1,...,c j ,...,c m}, for the jth block, using secure asymmetric scalar-product-preserving encryption (sASPE), we get the following formula:

[0122] SK j ={S j ,M 1,j ,M 2,j}

[0123] Encrypt it into the following formula:

[0124]

[0125] The encrypted dimension is 2d, and the cluster encrypted index vector is obtained:

[0126]

[0127] S33: Each node on the index tree is represented as:

[0128] node=<nID,V,left,right,CID / FID>

[0129] Among them, nID is the identifier of the node Node, left and right are pointers to the left child node and the right child node, CID is the medical classification cluster identifier corresponding to the node, FID is the corresponding medical Internet of Things multimodal data identifier, and FID only exists in leaf nodes and is 0 in non-leaf nodes. V stores specific features. In leaf nodes, V stores medical multimodal data, and in non-leaf nodes, it represents the bottom-up matching rules.

[0130] S34: Data encryption, using the symmetric encryption key κ to encrypt the document F i The plaintext data is encrypted into ciphertext data E i , and then encrypt the index And the ciphertext data E={E i}Upload to the cloud server. All data is first retrieved according to the cluster center, and then the forest structure is constructed for retrieval, that is, all data will be encrypted and indexed.

[0131] S35: Data update. After receiving the data update request from the data owner, the cloud server first calculates the cluster with the highest similarity among all clusters, and then enters the tree of the cluster to construct a node, which is expressed as:

[0132] node=<nID,I i,left,right,FID>

[0133] Insert the tree to update the index. In the clustered search tree, the leaf node represents the file identifier rather than the cluster identifier.

[0134] S36: The data user, as a hospital staff, passes the query to the edge server, which divides the query vector into m blocks of n-length sequences, q = {q1, q2, ..., q j ,...,q m}, for the jth block, use the ciphertext domain similarity hiding encryption algorithm with SK j ={S j ,M 1,j ,M 2,j} Encrypt it into Finally, we get the encrypted query

[0135] S37: After receiving the query request sent by the edge server After that, we use the block parallel retrieval method to perform parallel retrieval to obtain the best matching cluster, and then retrieve the top k best matching documents E in the subtree of the cluster. k , because of the influence of the encryption algorithm, the length of the algorithm vector is 2d, so the number of blocks is m, and the block length is 2n. And V of each tree node is a 2d-dimensional vector.

[0136] Specifically, the vectors of the query trapdoor and the root node of the index tree are divided into m blocks, and processed in parallel in different subtrees through multiple subqueries. First, a stack S is initialized and the root node is pushed into the stack. If the element is a leaf node and its relevance score with the query vector is higher than the lowest score kthscore in the current Top-k result, the list of the top k most relevant documents is updated, the document ID is inserted into it, and the document with the lowest score is removed. For non-leaf nodes, the relevance scores of its left and right child nodes with the query trapdoor are calculated. According to the relevance score, it is decided whether to push the child node into the stack and continue traversal. If the relevance score is higher than kthscore, the child node with high relevance is pushed into the stack, otherwise the top element of the stack is continued to be processed. After completing the traversal of the index tree, the Top-k related documents are selected from the merged results, and the final score check and update are performed to ensure that the returned documents are indeed the most relevant. Finally, the algorithm returns a list of the top k most relevant documents.

[0137] S38: The edge server receives the encrypted data set E returned by the cloud server k Afterwards, it is decrypted according to the decryption key to obtain the plaintext medical multimodal data and passed to the data user.

[0138] In summary, this paper proposes a multimodal encrypted retrieval algorithm for the medical Internet of Things with semantic feature fusion. Based on searchable encryption technology, a multimodal semantic feature extraction model is designed to realize encrypted retrieval of multimodal data in the medical Internet of Things. This paper proposes a semantic-aware multimodal data feature extraction method. Existing methods are based on traditional keyword extraction technology, which makes it difficult to perform semantic abstraction on text and image multimodal data. The proposed method enhances the semantic feature mining capability for multimodal data by designing a semantic feature extraction module for medical multimodal data, and replaces the traditional keyword retrieval mode with semantic feature retrieval.

[0139] The present invention proposes a low-overhead multimodal data encryption retrieval method. The existing encryption retrieval method has large search and storage overheads. The proposed method improves the existing ASPE encryption algorithm, protects the privacy information of the ciphertext domain similarity, improves the privacy protection level, and designs a parallel retrieval tree structure based on the block idea, so as to achieve low-overhead, privacy-protected semantic similarity fast retrieval.

[0140] The present invention also proposes a secure semantic search architecture for multimodal data in the medical Internet of Things. The traditional encrypted retrieval architecture is oriented towards single-modal data in the medical Internet of Things and cannot meet the search requirements for multimodal data. The proposed architecture is based on the idea of ​​edge-cloud collaboration, integrating the advantages of multimodal technology and searchable encryption technology. The edge is responsible for trapdoor generation and decryption of multimodal data, while the cloud is responsible for storage of multimodal data and semantic retrieval in a secret space. For the first time, it realizes low-latency, high-precision retrieval of multimodal data for users on the basis of protecting the privacy of users in the medical Internet of Things.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A semantically-aware cross-modal encrypted retrieval method, characterized in that: The following steps are involved: S1: Use the pre-trained semantic-aware multimodal feature extraction method to extract features from the query request and obtain multimodal features; S2: Using the similarity-hidden encrypted retrieval algorithm, an encrypted query trap is generated for the multimodal query features and uploaded to the cloud; S3: Cross-modal data encrypted retrieval: Use a parallel balanced binary tree index structure to perform similarity retrieval of the ciphertext domain in the cloud, sort the similarities using semantic relevance metrics, and return the retrieval results.

2. According to claim 1, a semantically-aware cross-modal encrypted retrieval method is characterized in that: The S1 specifically includes the following steps: S11: Model pre-training, the image encoder is initialized with the ResNet50 model, and the text encoder is initialized with PubMedBERT parameters; S12: Add a nonlinear mapping head to map the feature vector generated by the encoder to a uniform length, and use a nonlinear activation function to enable the network to learn complex feature representations; S13: Optimize the loss function by maximizing the inner product of image-text pairs; S14: Construct N positive samples through N image-text pairs, N 2 -N negative samples are used for training, where the positive samples are the corresponding image-text pair feature vectors, and the negative samples are the non-corresponding image-text pair feature vectors; S15: Use cosine similarity sim to measure the correspondence between image-text pairs. The larger the sim, the stronger the correspondence. S16: The training task is the optimization target, which is expressed as the following formula: Among them, I i is the feature vector of the i-th image, T j is the feature vector of the jth text; S17: The loss function is the cross entropy contrast loss function, expressed as the following formula: in, is the image-to-text contrast loss, is the contrast loss from text to image, N is the number of samples in the batch, and τ is the temperature parameter; S18: Assume that the image feature vector with multimodal data is f i =(f i,1 ,f i,2 ,...,f i,l ), the corresponding text feature vector is t i =(t i,1 ,t i,2 ,...,t i,l ), both of which are l-dimensional in length; the features of the corresponding image text are concatenated into a 2l-dimensional feature vector, which satisfies the following formula: f fusion_i =(t i ,f i )=(t i,1 ,t i,2 ,...,t i,l ,f i,1 ,f i,2 ,...,f i,l ) Among them, f fusion_i Fusion features preserved for multimodal semantic details; S19: Query image vector q i =(q i,1 ,q i,2 ,...,q i,l ) and text feature vector q t =(q t,1 ,p t,2 ,...,p t,l ), when generating a query trapdoor, the edge server concatenates the text and image feature vectors according to the query request; when it comes to unimodal queries, they are aligned by zero-padding.

3. According to claim 1, a semantically-aware cross-modal encrypted retrieval method is characterized in that: The S2 specifically includes the following steps: S21: Generate a key sk=(S,M1,M2) according to the security parameter λ, where M1 and M2 are d-dimensional random reversible matrices, and S is a d-dimensional random 0 / 1 sequence; assume that P is the set of vectors to be compared, and q is the query vector with a dimension of d; S22: Noise addition, for vector p in set P i , perform normalization, add noise to the feature vector, and expand it to d+2 dimensions; where the d+1th dimension is a random number δ, and the d+2th dimension is 1; S23: Normalize the query vector q, and then expand it to d+1 dimensions, where the d+1th dimension is 1; the total d+1-dimensional vector is perturbed by a random number β, 0<β<1, and the perturbed vector is {βq,β}, and then a random number c is added to the d+2 dimension, and finally: S24: splitting the vector according to the splitting rule of the random sequence S; If the i-th bit of S is 0, that is, S[i] = 0, then take in is a random number; when the i-th bit is 1, that is, S[i] = 0, in, and is a random number, S25: Use the dot product method to compare the similarity of query q in set P, which is expressed as the following formula: Score(Enc(p i ),Enc(q))=β(p i ·q+δ)+c Among them, Enc(p i ) is the index of the ciphertext state, Enc(q) is the query of the ciphertext state, p i q is the inner product similarity between the medical multimodal data feature vector and the query feature vector, δ is a random number, and c is a random number; S26: retain the encrypted function relationship of the same keyword query; through noise perturbation, use the randomness of δ and c to convert the similarity measurement into a size relationship; S27: When querying, sorting and retrieval are performed by measuring the similarity between different medical multimodal data index vectors p1 and p2 in the encrypted state and the query q.

4. According to claim 1, a semantically-aware cross-modal encrypted retrieval method is characterized in that: The S3 specifically includes the following steps: S31: Key generation, calculate m=d / n as the number of blocks, given the security parameter λ, generate the symmetric encryption key κ and the index encryption key SK, expressed as the following formula: SK=(S,M1,M2) Where S = {S1, S2, ..., S m },(S j ∈ R {0,1} d ), M1={M 1,1 ,...,M 1,j ,...,M 1,m },M2={M 2,1 ,...,M 2,j ,...,M 2,m },M 1,j and M 2,j is a d-dimensional random reversible matrix, and m≠d; S32: Index construction, according to the cluster center vector after clustering, it is divided into m blocks, each block is n in length, and each cluster representative is divided into C i ={c1,...,c j ,...,c m }, for the jth block, using secure asymmetric scalar product encryption technology, we get the following formula: EN j ={S j ,M 1,j ,M 2,j } Encrypt it into the following formula: The encrypted dimension is 2d, and the cluster encrypted index vector is obtained: S33: Each node on the index tree is represented as: node=<nID,V,left,right,CID / FID> Among them, nID is the identifier of the node Node, left and right are pointers to the left child node and the right child node, CID is the medical classification cluster identifier corresponding to the node, FID is the corresponding medical Internet of Things multimodal data identifier, and FID only exists in leaf nodes and is 0 in non-leaf nodes; V is to save specific features. In leaf nodes, V saves medical multimodal data, and in non-leaf nodes, it represents the bottom-up matching rules; S34: Data encryption, using the symmetric encryption key κ to encrypt the document F i The plaintext data is encrypted into ciphertext data E i ; Encrypt index I F = {I i } and ciphertext data E={E i }Upload to the cloud server; retrieve the above data according to the cluster center; construct a forest structure retrieval to encrypt the above data and construct a corresponding index; S35: Data update. After receiving the data update request from the data owner, the cloud server first calculates the cluster with the highest similarity among all clusters, and then enters the tree of the cluster to construct a node, which is expressed as: node=<nID,I i ,left,right,FID> Insert the tree to update the index. In the clustered search tree, the leaf node represents the file identifier rather than the cluster identifier. S36: The data user, as a hospital staff, passes the query to the edge server, which divides the query vector into m blocks of n-length sequences, q = {q1, q2, ..., q j ,...,q m }, for the jth block, use the ciphertext domain similarity hiding encryption algorithm with SK j ={S j ,M 1,j ,M 2,j } Encrypt it into Finally, we get the encrypted query S37: After receiving the query request sent by the edge server After that, we use the block parallel retrieval method to perform parallel retrieval to obtain the best matching cluster, and then retrieve the top k best matching documents E in the subtree of the cluster. k ; At this time, the length of the algorithm vector is 2d, so the number of blocks is m, and the block length is 2n; and V of each tree node is a 2d-dimensional vector; S38: The edge server receives the encrypted data set E returned by the cloud server k Afterwards, it is decrypted according to the decryption key to obtain the plaintext medical multimodal data and passed to the data user.

5. According to claim 3, a semantically-aware cross-modal encrypted retrieval method is characterized in that: The S24 further comprises the following steps: After the vector is split, it is encrypted according to the random reversible matrices M1 and M2, which is expressed as the following formula:

6. A semantically-aware cross-modal encrypted retrieval method according to claim 3, characterized in that: The ranking search in S27 is expressed as the following formula: Score(Enc(p2),Enc(q))-Score(Enc(p1)Enc(q))=β(p2·q-p1·q) Among them, β is only related to the query, and different queries have different β values. Therefore, this improved scheme can resist backward attacks and prevent the original similarity results from being leaked between different queries.

7. The semantically-aware cross-modal encrypted retrieval method according to claim 4, characterized in that: The S37 is specifically: Divide the query trapdoor and the vector of the index tree root node into m blocks and process them in parallel in different subtrees through multiple subqueries; Initialize a stack S and push the root node into the stack. If the element is a leaf node and its relevance score to the query vector is higher than the lowest score kthscore in the current Top-k results, update the list of the top k most relevant documents, insert the document ID into it, and remove the document with the lowest score. For non-leaf nodes, calculate the correlation scores between its left and right child nodes and the query trapdoor; decide whether to push the child node into the stack and continue traversing based on the correlation score; if the correlation score is higher than kthscore, push the child node with high correlation into the stack, otherwise continue to process the top element of the stack; After completing the traversal of the index tree, select the top-k relevant documents from the merged results, and perform a final score check and update to ensure that the returned documents are the most relevant; Returns a list of the top k most relevant documents.

Images