Lightweight and privacy-protected cross-modal retrieval method in Internet of Things environment

Abstract

The invention relates to a lightweight and privacy protection cross-modal retrieval method in an Internet of Things environment. The method comprises the steps that S1, a data owner generates and sends a secret key to a data user; s2, training the cross-modal feature extraction model; s3, the data owner inputs data to be uploaded into the trained model, and the obtained data features are converted into binary vectors; s4, encrypting the data obtained in the S3, generating a quadtree index, encrypting data to be uploaded, and sending the data and the quadtree index to a cloud server; s5, the data owner generates and sends AAI and digital signature information to the data user; s6, the data user converts the query data into binary vectors, encrypts the binary vectors and sends the binary vectors to the cloud server; s7, the cloud server generates a query result and integrity verification certification information and sends the query result and the integrity verification certification information to the data user; and S8, the data user verifies the query result, and when the query result is correct, the cloud server sends the encrypted data to the data user. According to the method, the complexity of a model for extracting data features is reduced while privacy protection is carried out on the data.

Description

Technical Field

[0001] The present invention relates to a cross-modal retrieval method, in particular to a lightweight and privacy-preserving cross-modal retrieval method in an Internet of Things environment. Background Art

[0002] At present, the Internet of Things (IoT) has been widely used in many fields such as manufacturing, transportation, energy, and healthcare. Therefore, physical devices deployed through the IoT will generate a large amount of multi-source heterogeneous data. For example, in the early warning system of industrial sewage treatment equipment, there are many equipment attribute description data, such as temperature, clean water storage, and machine speed. Self-driving cars continuously monitor the surrounding environment by using multiple sensors, and the on-board cameras collect a large amount of image data. Due to the high storage and communication costs of IoT devices, image data needs to be outsourced to cloud servers for storage and management. However, outsourcing makes the data out of the control of the owner, resulting in some huge challenges in the security of the IoT ecosystem, such as privacy leakage and poor availability, which seriously hinders the deployment of the IoT ecosystem.

[0003] Traditional encryption algorithms (such as AES, RSA, homomorphic encryption (HE), comparative encryption (CE), secure multi-party computation, etc.) (SMC) can ensure security, but they limit the availability of image data and have high computational overhead, making them inapplicable to IoT devices with limited resources. Therefore, searchable encryption technology provides a viable option for the storage and utilization of image text data in the IoT. This technology can perform retrieval in the ciphertext domain, protecting data privacy while not affecting data availability. However, most existing SE solutions may pose challenges when directly applied in the IoT environment. For example, keyword-oriented retrieval technology cannot meet the requirements of ciphertext retrieval of large-scale multi-source heterogeneous data in the IoT. Single-modality data retrieval queries are difficult and inefficient.

[0004] The main challenge in retrieving multi-source heterogeneous data is that different modal data are represented in different forms, that is, the two data are in different distribution spaces, and similarity cannot be directly measured. Cross-modal retrieval technology can measure the visual semantic similarity between images and text descriptions. However, the current existing technologies do not consider the issue of user data privacy protection. More importantly, whether it is Faster R-CNN for extracting image features or BERT for extracting text features, the number of parameters in the model is in the millions or above, so it cannot be deployed in IoT terminals with limited resources.

[0005] As a link between vision and language, cross-modal image-text retrieval has been widely studied in recent years. However, existing studies usually design complex machine learning models to improve retrieval accuracy and do not consider the privacy protection of images and texts, which is not suitable for lightweight IoT environments. Summary of the invention

[0006] The purpose of the present invention is to provide a lightweight and privacy-preserving cross-modal retrieval method in an Internet of Things environment to solve the complex problems of existing feature learning models.

[0007] The object of the present invention is achieved in that:

[0008] A lightweight and privacy-preserving cross-modal retrieval method in an Internet of Things environment comprises the following steps:

[0009] S1. The data owner generates a key for encrypting a binary vector of the data to be uploaded and a key for encrypting a binary vector of the query data; and sends the key for encrypting the binary vector of the query data to the data user;

[0010] S2. Training a cross-modal feature extraction model;

[0011] S3. The data owner inputs the data in the dataset to be uploaded into the trained cross-modal feature extraction model to obtain data features; performs binary code conversion and expansion on the data features to generate a binary vector. The dataset to be uploaded includes the data and the identifier corresponding to the data;

[0012] S4. Encrypt the binary vector according to the key used to encrypt the binary vector of the data to be uploaded, generate a quadtree index using the quadtree index structure and the encrypted binary vector, and encrypt the data to be uploaded; send the quadtree index and the encrypted data to be uploaded to the cloud server;

[0013] S5. The data owner generates integrity verification auxiliary information AAI and a signature private key, and generates digital signature information corresponding to each data according to the signature private key and identifier; the data owner sends the AAI to the data user, and sends the identifier and digital signature information of each data to the cloud server;

[0014] S6. The data user inputs the query data into the trained cross-modal feature extraction model in step S3 to obtain data features, converts and expands the data features into binary codes, and generates a binary vector corresponding to the query data; encrypts the binary vector corresponding to the query data according to the key used to encrypt the feature vector of the query data to generate a query vector, and sends the query vector to the cloud server;

[0015] S7. The cloud server calculates the similarity between the query vector and each non-leaf node of the quadtree index from top to bottom, and generates a query result; the cloud server generates integrity verification proof response information based on the query result and the digital signature information, and sends the integrity verification proof response information and the query result to the data user;

[0016] S8. The data user verifies the query result based on the AAI and integrity verification auxiliary information; when the query result is correct, the data user sends an encrypted data request to the cloud server, and the cloud server sends the encrypted data to be uploaded corresponding to the query result to the data user.

[0017] Furthermore, the specific method of training the cross-modal feature extraction model in step S2 is:

[0018] In the cross-modal feature extraction model, the pre-trained Faster R-CNN is used as the teacher model for extracting image features, the Transformer is used as the student model of Faster R-CNN, the pre-trained BERT is used as the teacher model for extracting text features, and the LSTM is used as the student model of BERT. The student model uses the output of the teacher model as a reference for training, and the loss function is used to optimize the training process.

[0019] Faster R-CNN: Compared with its predecessor R-CNN, Faster R-CNN integrates the region proposal in the object detection process with the feature extraction and classification processes, thus achieving end-to-end training at a faster speed.

[0020] Transformer encoder: It improves the shortcomings of CNN-based image retrieval, such as large amount of training data, long model training time, and insufficient use of supervisory information. It has excellent computational efficiency and scalability and can handle large amounts of data.

[0021] BERT: It is an advanced natural language processing technology that can replace the traditional Word2Vec method with excellent results. Compared with other language models, BERT uses a deep language representation encoded with a bidirectional Transformer, which more comprehensively captures the bidirectional relationship in the sentence. This bidirectional structure can obtain information in both the front and back directions of the text, and can better utilize the contextual information of the corpus during the training process. It also uses a Transformer structure with a self-attention mechanism, which can learn according to different contexts and has a certain error correction capability.

[0022] Furthermore, the calculation formula of the loss function is:

[0023] L stu =χL+(1-χ)L dis

[0024] Among them, L stu is the loss function, χ is the balance parameter, L is the contrast loss function, L dis The distillation loss function is calculated as the difference between the output of the student model and the teacher model;

[0025] The calculation formula of the contrast loss function is:

[0026] L=L a +L r

[0027] L r =max(0,δ-S(I,T)+S(I,T l -))

[0028] +max(0,δ-S(I,T)+S(I v -,T))

[0029]

[0030] Among them, L a is the intra-model consistency comparison loss, L r is the inter-modality consistency contrast loss; δ is the edge parameter, S(·) is the similarity function; I is the image sample, T is the text sample, v- is the hard negative sample index of the image modality, l- is the hard negative sample index of the text modality, and σ is the relaxation variable;

[0031] The calculation formula of the distillation loss function is:

[0032]

[0033] Among them, x i is the output of the teacher model, y i is the output of the student model.

[0034] Furthermore, the specific method of generating the quadtree index in step S4 is:

[0035] The encrypted binary vector is stored in the leaf node, and an internal node is generated for every four child nodes. The binary vector corresponding to each four child nodes generates a representative vector stored in the internal node until a root node and a binary vector stored in the root node are generated to obtain a quadtree index; when the last group is less than four nodes, a virtual node and a binary vector stored in the virtual node are filled, and the binary vector stored in the virtual node is all zero.

[0036] Furthermore, the specific method of generating the binary vector stored in the internal node is:

[0037] If a binary vector stored in a non-leaf node is generated from a binary vector stored in a leaf node, the average value of the binary vectors corresponding to the four added leaf nodes at each position is calculated to generate an average vector, and the positions in the average vector that are greater than a predefined threshold are assigned a value of 1, and the positions that are less than the predefined threshold are assigned a value of 0, thereby generating a binary vector stored in a non-leaf node;

[0038] If a binary vector stored in a non-leaf node is generated by a binary vector stored in a non-leaf node, as long as there is one 1 in the same position of the binary vectors corresponding to the four non-leaf nodes, the corresponding position of the generated binary vector is 1; if the binary vectors corresponding to the four non-leaf nodes are all 0 in the same position, the corresponding position of the generated binary vector is 0.

[0039] Furthermore, the specific method of generating the query result in step S7 is:

[0040] The cloud server first calculates the similarity between the query vector and each non-leaf node, and selects the non-leaf nodes whose similarity is greater than the similarity threshold; then calculates the similarity between the query vector and all leaf nodes corresponding to the selected non-leaf nodes, and finally sorts them according to the similarity values, and selects the data identifiers corresponding to the first k leaf nodes as the query results.

[0041] Furthermore, the specific method for the data owner to generate the key for encrypting the binary vector and the key for encrypting the feature vector of the query data is:

[0042] S1a-1. The data owner generates system public parameters based on the security parameter λ, and the system public parameters include G1, G2, e, g, p, p pub , w and H, where G1 and G2 are multiplicative cyclic groups of order p, G2 = G1 × G1, g is a generator, and G1·p pub =g s , s is in Z p * is selected in G1, w is selected in G1, Z p1 Z p The range of values ​​after the hash function, G1 is from Z p1 taken out from

[0043] S1a-2. Generate two random matrices M and M -1 , where M, M -1 ∈Z (2d)×(2d) , M -1 is the inverse matrix of M, d is the dimension of the eigenvector; generate random vectors α and β, where α={α1,α2,…,α d-1}, β={β1,β2,…,β d-1}, α1, α2, …, αd-1 ,β1,β2,…,β d-1 ∈Z p2 ; Generate a random integer γ, a random positive number ε, a random number θ and two random integer error vectors ξ o and u ; Calculate W = θ·M -1 ;

[0044] S1a-3. The key key that will be used to encrypt the binary vector o ={M,α,γ,ξ o}Sent to the data owner, the key used to encrypt the feature vector of the query data u = {W,β,ε,γ,ξ u}Sent to data user.

[0045] Furthermore, the specific method for the data owner to generate integrity verification auxiliary information and signature private key is:

[0046] Data owner is in Z p Uniformly select η, κ and τ from the equation; calculate the seed information ψ1 = η.κ.τ, ψ2 = η 2 .κ 2 .τ,…,ψ |M| =η |M| .κ |M| .τ, and integrity verification auxiliary information AAI = {η,κ,τ,H(ψ1),H(ψ2),…,H(ψ |M| )}, where H is the hash function and each image corresponds to a ψ i ; Generate signature private key K A , where K A =(r+s)H(ID o ), where r is the p *Selected from, ID o is the identification code of the data owner; and selects the identifier id of each data in the data set j , calculate the digital signature information corresponding to each data

[0047] Furthermore, the specific method for the cloud server to generate the integrity verification proof response information is as follows:

[0048] Cloud Server from Z p * randomly select c and calculate e=w c and Calculate the aggregate signature The cloud server generates integrity verification proof response information {μ,e,Sig}.

[0049] Furthermore, the specific method for data users to verify the query results is:

[0050] S8-1. Data users calculate φ based on AAI. The calculation formula for φ is:

[0051] S8-2. Using the value of φ to judge Is it true? If so, the query result is complete. If not, the query result is incomplete.

[0052] In order to ensure the integrity of query data, the present invention uses the idea of ​​aggregate signature to design a lightweight query result verification method, which reduces the verification communication and computing overhead and query response time; in order to adapt to the limited resource devices in the Internet of Things, a lightweight and privacy-preserving image-text matching method based on knowledge distillation is proposed. This method uses a secure KNN algorithm based on error learning to encrypt the binary encoding of the feature vector, ensuring the privacy of the underlying vector under the linear analysis attack model. The present invention encrypts the data to be uploaded and uses indexes for querying, which improves the privacy of the data.

[0053] In order to retrieve large-scale multi-source heterogeneous data in the Internet of Things, the present invention proposes a quadtree index structure based on a hybrid representative vector, which uses different representative vector generation methods for leaf nodes and internal nodes to improve retrieval efficiency and accuracy. In order to ensure the integrity of the query data, a lightweight query result verification method is designed using aggregate signature technology. The more representative feature vectors in images and texts are extracted through the ML model, the semantic information contained in the text query and the image is fully mined, and the feature vectors of different modalities are semantically aligned to perform comprehensive matching. The present invention verifies the experimental results through a large number of experiments under real-world data sets.

[0054] The present invention trains a lightweight student model through knowledge distillation to reduce the model complexity, extract the feature vectors of images and texts and embed them into the same semantic space. More importantly, in order to improve the retrieval accuracy of the student model, a new training indicator, intra-model consistency comparison loss, is proposed to ensure the semantic consistency of images and texts in the common embedding space and improve the accuracy of data query. In addition, a new quadtree index structure is proposed based on mixed representative vectors to effectively reduce the retrieval time overhead. In this structure, the feature vectors and representative vectors of each image and text are encrypted using a secure KNN algorithm based on LWE to achieve image-text matching in a large-scale ciphertext environment. Finally, the security of the present invention is evaluated through detailed formal analysis and the effectiveness of the present invention is verified in two real-world data sets (i.e., COCO and Flickr30k) through a large number of experiments. The experimental results show that the size of the student model is reduced by 39.46% to 53.29% compared with the teacher model, but the retrieval accuracy can reach 88% to 97% of the teacher model. The retrieval accuracy of the student model and the teacher model is significantly improved compared with the latest scheme. Using the proposed quadtree for retrieval, the retrieval time is reduced by 3.7% to 54.2% compared with the latest solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a system framework diagram.

[0056] Figure 2 It is a flow chart of the feature extraction method.

[0057] Figure 3 It is a flowchart for generating a quadtree index.

[0058] Figure 4 It is the search effect diagram of the present invention.

[0059] Figure 5 (a) is a comparison of the retrieval accuracy of the student model and the teacher model when the relaxation variable σ is equal to different values ​​when using text to retrieve images; (b) is a comparison of the retrieval accuracy of the student model and the teacher model when the relaxation variable σ is equal to different values ​​when using images to retrieve text.

[0060] Figure 6 (a) is a comparison of the retrieval accuracy of the student model and the teacher model when the balance parameter χ is equal to different values ​​when using text to retrieve images; (b) is a comparison of the retrieval accuracy of the student model and the teacher model when the balance parameter χ is equal to different values ​​when using images to retrieve text.

[0061] Figure 7 It is a comparison chart of the retrieval time of the index structure of the present invention and other index structures.

[0062] Figure 8This is a comparison chart of the construction time of the index structure of the present invention and other index structures. DETAILED DESCRIPTION

[0063] The present invention is further described below.

[0064] like Figure 2 As shown, the present invention provides a lightweight and privacy-preserving cross-modal retrieval method in an Internet of Things environment, which specifically includes the following steps:

[0065] A lightweight and privacy-preserving cross-modal retrieval method in an Internet of Things environment comprises the following steps:

[0066] S1. The data owner generates a key for encrypting a binary vector of the data to be uploaded and a key for encrypting a binary vector of the query data; and sends the key for encrypting the binary vector of the query data to the data user.

[0067] like Figure 1 As shown in the figure, in the IoT system, there are data owners Do, data users Du, and cloud servers CSP. Do refers to the devices in the IoT system. For example, smart terminals such as mobile phones in the medical IoT, and gateways in the industrial IoT. Do is responsible for collecting image text data and extracting features, then aligning the relevant image-text, encrypting it and uploading it to CSP. Do also sends a key to the authorized user for subsequent generation of a search token.

[0068] CSP: Responsible for storing encrypted images, files, and indexes, providing search services for users, and generating query result verification credentials for DU verification.

[0069] Du: Du who has access requirements applies for authorization. After authentication, a secure channel is established between the two parties. Do sends the key to Du through the secure channel. After receiving the key, Du generates the corresponding search token and sends it to CSP. CSP performs a cross-modal search operation and returns the search results to Du, who then verifies the results.

[0070] The data owner generates the system public parameters G1, G2, e, g, p, p according to the security parameter λ pub , w and H, where G1 and G2 are multiplicative cyclic groups of order p, G2 = G1 × G1, g is a generator; G1·p pub =g s , s is in Z p * is selected in G1, w is selected in G1; Z p1 Z p The range of values ​​after the hash function, G1 is from Z p1 Taken out, Z p2 Zp Another range of values ​​after the hash function, the system public parameters are sent to all participants.

[0071] The data owner generates two random matrices M and M -1 , where M, M -1 ∈Z (2d)×(2d) , M -1 is the inverse matrix of M, d is the dimension of the eigenvector; and generates random vectors α and β, where α={α1,α2,…,α d-1}, β={β1,β2,…,β d-1}, α1, α2, …, α d-1 ,β1,β2,…,β d-1 ∈Z p2 ; Generate a random integer γ, a random positive number ε, a random number θ and two random integer error vectors ξ o and u , where γ∈Zp1, p1>>p2, and γ>>2|max(ξ u )|, where ξ u ∈Z 2d p2 ; Calculate W = θ·M -1 ; The key key o ={M,α,γ,ξ o} is sent to the data owner to encrypt the binary vector; the key key u = {W,β,ε,γ,ξ u}Sent to the data user to encrypt the feature vector of the query data.

[0072] S2. Train the cross-modal feature extraction model.

[0073] The present invention adopts the method of knowledge distillation to retrain two student models with simple structures through the teacher model. During training, the student model will use the output of the teacher model as a reference, so as to achieve an output similar to that of the teacher model when the model parameters are much smaller than those of the teacher model. In addition, in order to semantically align the feature vectors of images and texts, a new loss function is given, which effectively improves the retrieval efficiency and accuracy. In order to ensure the privacy of the data, the binary encoding of the feature vector is encrypted using the secure KNN algorithm based on error learning to ensure the privacy of the underlying vector under the linear analysis attack model. Figure 2 It includes three stages: feature extraction, encryption, and retrieval.

[0074] The present invention searches for cross-modal data. Since images and texts are two heterogeneous modal information, there is a huge semantic gap between them. In order to fully understand the semantic information in the two modalities and accurately return matching results, the present invention uses a transformer-based structure for representation learning and obtains global and local feature representations at the same time.

[0075] For the image set, in order to obtain more descriptive information of the visual content of the image area and adapt to the resource-constrained environment of the IoT device, the present invention uses the pre-trained Faster R-CNN as the teacher model. Faster R-CNN will extract the object features and bounding box information in the image. Faster R-CNN also contains four layers of the same standard Transformer encoder. The student model of Faster R-CNN is a layer of Transformer encoder. Faster R-CNN passes the output features to a layer of Transformer encoder (i.e., the student model), thereby further processing the feature representation of each region to capture the semantic relationship and contextual information between regions. In the present invention, in order to compress the model volume while retaining the retrieval accuracy. The present invention reduces the four-layer Transformer encoder in the teacher model to 1 layer, thereby training a new student model. During training, the student model will use the output of the teacher model as a reference, so as to achieve an output similar to the teacher model when the model parameter amount is much smaller than the teacher model.

[0076] For the text set, the present invention uses the widely used pre-trained model BERT as the teacher model, and uses LSTM as the student model of BERT to extract text semantic information at the sentence and word levels. In the present invention, the structure of LSTM is usually simpler than that of BERT, but it has a retrieval accuracy similar to that of BERT. Therefore, the present invention uses LSTM as the student model.

[0077] In order to ensure the semantic consistency of image features and text features in the common embedding space, the present invention widely uses triple ranking loss, which can constrain the distance between feature vectors of different modalities. For example, for an image-text pair (I, T) to be matched, I and T represent the image and text respectively, and the image or text can be used as an anchor to obtain a hard negative sample pair. With image I as the anchor, the present invention can find a hard negative image sample I v- and its corresponding text sample T v- This gives us two hard negative pairs (I v- , T v- ) and (I l- , T l- ), where v- is the hard negative sample index of the image modality and l- is the hard negative sample index of the text modality. Inter-modality consistency contrast loss Lr The calculation formula is:

[0078] L r =max(0,δ-S(I,T)+S(I,T l -))

[0079] +max(0,δ-S(I,T)+S(I v -,T))) (1)

[0080] Among them, δ is the edge parameter and S(·) is the similarity function used to calculate the similarity of image-text pairs.

[0081] However, the standard triple ranking loss has the problem of only performing comparative learning between different modalities, which only constrains the distance between modalities but not the distance within the modality.

[0082] Therefore, the present invention proposes a new intra-modal consistent contrast loss. Specifically, in order to enhance intra-modal consistency, the distance between samples in the same modality should be reduced, which helps to reduce the difference between modalities and the influence of noise. At the same time, the main purpose of the intra-modal loss is to ensure that the distance between two pairs of samples in the same modality is consistent. a The calculation formula is:

[0083]

[0084] Among them, σ is the slack variable, I l- and T l- is a hard negative sample pair obtained with text T as the anchor point.

[0085] In addition, in order to consider both the inter-modal loss and the intra-modal loss, the final contrast loss function is defined as L r and L a Through the final loss function, the distance between matching image-text pairs is shortened, the distance between mismatching image-text pairs is increased, and at the same time, the distance between samples in the same modality is shortened and becomes consistent. The calculation formula of the contrast loss function L is:

[0086] L=L a +L r (3)

[0087] Among them, L is the contrast loss function, L a is the intra-model consistency comparison loss, L r is the inter-module consistency comparison loss.

[0088] When performing knowledge distillation, the student model fits the output of the teacher model through a loss function. The present invention uses mean square error loss as a loss function to calculate the difference between the output of the student model and the output of the teacher model.

[0089] Assume X = {x0, x1, ..., x n} and Y={y0,y1,...,y n} are the outputs of the teacher model and the student model respectively. When calculating the loss L of the student model stu , the distillation loss is weighted summed with the loss L of the student model during training. L is the loss of the student model itself during training, L dis It is the distillation loss calculated by the difference between the output of the student model and the teacher model, and α is used to proportionally learn the knowledge transferred by the teacher model, that is, the balance parameter.

[0090] L stu =αL+(1-α)L dis (4)

[0091]

[0092] Among them, L stu is the loss function, L dis is the distillation loss function calculated by the difference between the output of the student model and the teacher model, x i is the output of the teacher model, y i is the output of the student model.

[0093] S3. The data owner inputs the data in the dataset to be uploaded into the trained cross-modal feature extraction model to obtain data features; performs binary code conversion and expansion on the data features to generate binary vectors.

[0094] The data set to be uploaded includes data and an identifier corresponding to the data, and the original data in the data set to be uploaded is an image or text.

[0095] In the cross-modal feature extraction model, the Transformer encoder extracts image features, and the LSTM extracts text features. When the data set to be uploaded contains one data, the original data of the data set to be uploaded contains text or images; when the data to be uploaded contains at least two data to be uploaded, the original data of the data set to be uploaded contains text and / or images. For images IM1, IM i ,…,IM |IM| , for each image IM i , the detector generates n image proposals describing the object instance, and a feature vector f corresponding to each proposal. i ∈R d and the bounding box vector bi ∈R4. The image proposals for each image can be represented as V = {v0,v1,v2,…,v n}, where v j =[f i ,b i ]∈R d+4 , j∈{1,n}. After using each element in V as the input of the cross-modal feature extraction model, the cross-modal feature extraction model outputs the learned image features V'={v0',v1',v2',…,v n '}.

[0096] For a text set, for a given sentence containing w words, a total of w+1 tokens will be generated for the entire sentence and the words in it. Define the text representation as t0,t1,…,t w , where w is the number of words in the sentence. The text is extracted through the cross-modal feature extraction model, and the output of the cross-modal feature extraction model is the text feature t'={t0',t1',t2',…,t n '}.

[0097] For the image features V'={v0',v1',v2',…,v n '} is converted into a short binary code bv = {bv0, bv1, bv2, ..., bv n}, and the text feature t'={t0',t1',t2',…,t n '} is converted to bt = {bt0, bt1, bt2, ..., bt n}.

[0098] The core idea of ​​converting to similarity-preserving binary code is to find the rotation of the zero-centered data so that the quantization error of mapping these data to the vertices of the zero-centered binary hypercube is minimized. Before using ITQ, the data is first reduced in dimension using PCA principal component analysis. The feature vectors of images and texts are encrypted in the same way.

[0099] The data owner receives the key o The α in the image feature binary code and the text feature binary code are expanded respectively. For 0≤i≤n, each code in the image binary code is expanded to generate the image binary vector bv i '={bv i0 ,…,bv id-1 ,α1,α2,…,α d}; Expand each code in the text binary code to generate a text binary vector bt i '={bt i0 ,…,btid-1 ,α1,α2,…,α d}. Among them, α1α2,…,α d ∈Z p2 is a random number chosen for each binary code.

[0100] S4. Encrypt the binary vector according to the key used to encrypt the feature vector of the data to be uploaded, generate a quadtree index using the quadtree index structure and the encrypted binary vector, and encrypt the data to be uploaded; send the quadtree index and the encrypted data to be uploaded to the cloud server.

[0101] The data owner receives the key o Encrypt the binary vector. For the image, the encryption formula is:

[0102] E M (bv i )=(γbv i '+ξ o )M (6)

[0103] For text, the encryption formula is:

[0104] E M (bt i )=(γbt i '+ξ o )M (7)

[0105] like Figure 3 As shown, in order to achieve sublinear search complexity, the present invention designs a new quadtree index structure and a corresponding retrieval algorithm based on a mixed representative vector. First, the encrypted binary vector is stored in the leaf node of the index tree, and every four leaf nodes are grouped to generate an internal node, and the binary vectors stored in the four leaf nodes generate the binary vectors stored in the generated internal nodes; then the four internal nodes are grouped to generate the internal nodes of the next layer, and the binary vectors stored in the four internal nodes generate the binary vectors stored in the internal nodes of the next layer; finally, the root node and the binary vector stored in the root node are generated.

[0106] When the binary vector stored in the leaf node generates the binary vector stored in the internal node, the average value of the binary vectors of the leaf nodes at the same position is calculated to obtain the average vector, such as Figure 3 The values ​​of the first position of the binary vector in are 1, 0, 1, and 1 respectively, then the value of the first position in the average vector is 0.75. The median value of the average vector is greater than the predefined threshold Th RV The position with a value of 1 is assigned, and the position with a value less than a predefined threshold is assigned to 0. Assume that Th RV=0.7, indicating that more than 70% of the features of the image at the current position are 1. RV Setting it to a lower or higher value will cause the representative vector to have too many "1" (lower value) or "0" (higher value) bits, which will affect the query results. RV The value of needs to be debugged through experiments.

[0107] When the binary vector stored in the internal node (non-leaf node) generates the binary vector stored in the internal node, the binary vectors are superimposed bit by bit to generate the binary vector of the next layer. As long as there is a 1 in one position, the corresponding position of the generated binary vector is 1; when all are 0, the generated binary vector is 0. For example, Figure 3 When the internal node N 1,0 ,N 1,1 ,N 1,2 and N 1,3 The binary vector generated is stored in node N 2,0 If the first bit of the binary vector has a 1, the first bit of the generated binary vector is 1.

[0108] The data owner generates the data encryption key K O , according to K O The data to be uploaded is encrypted, and the quadtree index and encrypted data are sent to the cloud server.

[0109] For sublinear retrieval of multi-source heterogeneous data, the present invention designs a quadtree index structure based on a hybrid representative vector, uses different representative vector generation methods for leaf nodes and internal nodes, and improves retrieval efficiency and accuracy.

[0110] S5. The data owner generates integrity verification auxiliary information AAI and a signature private key, and generates digital signature information corresponding to each data based on the signature private key and identifier; the data owner sends the AAI to the data user, and sends the identifier and digital signature information of each data to the cloud server.

[0111] Data owner DO from Z p Randomly and uniformly select η, κ and τ from the equation, and calculate the seed information ψ1 = η.κ.τ, ψ2 = η 2 .κ 2 .τ,…,ψ |M| =η |M| .κ |M| .τ, each data has a corresponding Ψ i , and integrity verification auxiliary information AAI = {η,κ,τ,H(ψ1),H(ψ2),…,H(ψ |M| )}; where H is a hash function.

[0112] The data owner generates a signature private key K A , where K A =(r+s)H(ID o ), where r is the p *Selected from, ID o The identifier of the data owner.

[0113] According to the system public parameters, signature private key K A And the identifier of each data is used to calculate the digital signature information corresponding to each data. The calculation formula is:

[0114]

[0115] Among them, Sig j Identifier id j The digital signature information corresponding to the data.

[0116] The data owner sends the AAI to the data user through a secure channel, j ,Sig j )}| (1<j<|M|) Upload to the cloud server and delete the backup records locally.

[0117] S6. The data user inputs the query data into the trained cross-modal feature extraction model in step S3 to obtain data features, converts and expands the data features into binary codes, generates a binary vector corresponding to the query data, and encrypts the binary vector corresponding to the query data according to the key used to encrypt the feature vector of the query data to generate a query vector (i.e. Figure 1 The query vector is sent to the cloud server.

[0118] The original data of the query data is an image or text.

[0119] The data user inputs the query image or query text into the trained cross-modal feature extraction model to obtain the query image feature or query text feature, q = {q0, q2, ..., q n}, where q i = {q i,0 ,q i,1 ,...,q i,d-1}, i∈[0,n]. Convert the obtained query image features or query text features into binary code b q = {b q0 ,b q1 ,...,b qn}, and then expand the binary code, for 0≤i≤n, to generate the query vector b qi '={{εb qi,0 ,εbqi,1 ,···,εb qi,d-1 ,β1,β2,···,β d}, and finally encrypt the query vector. The encryption calculation formula is:

[0120] E W (b q )=W(γ·b qi '+ξ u )(9)

[0121] Among them, E W (b q ) is the encrypted query vector.

[0122] S7. The cloud server calculates the similarity between the query vector and each non-leaf node of the quadtree index from top to bottom to generate the query result; the cloud server generates integrity verification proof response information based on the query result and digital signature information, and sends the integrity verification proof response information and the query result to the data user.

[0123] After receiving the query request, the cloud server calculates the query vector E from top to bottom W (b q ) and the similarity S between the binary vectors stored in non-leaf nodes iq , after finding the binary vectors whose similarity is greater than the similarity threshold, calculate E W (b q ) and the binary vector N stored in all child nodes of the non-leaf node storing the binary vector 0,i Finally, press S iq The values ​​of are sorted, and the identifiers of the data corresponding to the k nodes with the largest values ​​are selected as the query results, that is, RS = {id1,…,id k} and sent to the data user. Similarity S iq The calculation formula is:

[0124] S iq =N 0,i ·E W (b q )(10)

[0125] Among them, N 0,i is the binary vector stored at node (0,i).

[0126] When performing cross-modal retrieval, the encrypted vector of the query text is used to perform hierarchical retrieval from the index root node from top to bottom to find the most relevant k query results. The greedy algorithm is mainly used to find the most relevant node at each layer, and then visit all the child nodes of the node, and so on until the target node is found. In addition, the most relevant k query results can be found by backtracking. In short, this method mainly improves retrieval efficiency by finding the local optimal solution.

[0127] The cloud server calculates the values ​​of μ, β, and Sig, first in Z p * Randomly select c and calculate e=w c ,as well as Calculate the aggregate signature The cloud server sends the integrity verification proof response information {μ,e,Sig} to the data user.

[0128] S8. The data user verifies the query result based on the integrity verification proof response information and the integrity verification auxiliary information (i.e. Figure 1 Authentication in); when the query result is correct, the data user sends an encrypted data request to the cloud server, and the cloud server sends the encrypted data to be uploaded corresponding to the query result to the data user. When it is incorrect, the data user refuses to receive the encrypted data to be uploaded corresponding to the query result.

[0129] Data users are calculated based on AAI And use the value of φ to judge the integrity verification equation Is it true? If it is true, it means that the query result is complete. The calculation process of the verification equation is:

[0130]

[0131] Among them, q k is the kth element in the query result.

[0132] S9. Security verification and correctness verification.

[0133] S9-1. Security verification.

[0134] S9-1-1. Index and query privacy proof:

[0135] For the problem of learning with error (LWE): given multiple samples of a polynomial (x i ∈Z d p1 ,y i ∈Z p1 ) where the error term From some probability distribution, recovering the vector f with non-negligible probability is computationally infeasible.

[0136]

[0137] Under the known ciphertext attack model, the present invention (LP 2 CR-IoT) can protect index and query privacy.

[0138] Proof: Taking image features as an example, in the present invention, each extended vector b v ' is encrypted as:

[0139] E M (bv)=(bv'·γ+ξ o )·M (13)

[0140] Because b v ' and ξ are two vectors, and formula (13) can be converted into a two-dimensional vector point:

[0141] E M (bv)(k)=bv'·γ·M(k)+ξ o ·M(k) (14)

[0142] Among them, 1≤k≤2d, E M (bv)(k) is E M (bv) is the kth element, M(k) is the kth column of M. The present invention defines γ·M(k) as M'(k), defines ξ·M(k) as ξ', and then there is a 2d sample (M'(k), EM(bv)(k))

[0143] E M (bv)(k)=bv'·M'(k)+ξ o ',1≤k≤2d (15)

[0144] Therefore, from the encrypted vector EM(b v )Recovering bv' becomes solving the previous LWE problem. Solving the LWE problem is computationally infeasible for a polynomial-time adversary. In addition, M is a secret key that the attacker cannot obtain. M' cannot be exploited by the adversary either, and recovering bv' from the encrypted vector EM(bv) is more difficult than solving the LWE problem. q and its ciphertext EW(b q ), which can also be encrypted as a two-dimensional dot product:

[0145] E W (bq)(k)=W'(k)·q' T +ξ u ',1≤k≤2d (16)

[0146] where W(k) = γ·W(k), W(k) is the kth column of M, and ξ u '=W(k)·ξ u T , so recovering bq from EW(bq) can be reduced to given two-dimensional samples (W(k), E W (bq)(k)) to solve the LWE problem. Since W is a secret key, only the authorized user knows it, so the adversary cannot obtain it.

[0147] Therefore, if the LWE problem is hard, then recovering the index and query vectors from the corresponding ciphertext is computationally infeasible for a polynomial-time adversary. That is, under a known ciphertext attack model, the privacy of the index and query is well preserved.

[0148] Under a known background attack model, the present invention can protect index and query privacy.

[0149] Proof: In the known background attack model, the attacker has access to the ciphertext and can obtain some query vectors from some background information. We have proved that the attacker cannot recover the plaintext from the ciphertext of the index vector and the query vector. Now, an attacker with additional background knowledge may perform a linear analysis attack to try to recover the plaintext from the inner product calculation result. Specifically, given a query vector b q and any two index vectors N a ,N b , the attacker can construct an equation as

[0150]

[0151] There are 3d+1 unknowns in formula (17). Assuming the adversary obtains bq, the number of unknowns in formula (17) will be reduced to 2d+1. Linear analysis attack means that if the attacker can obtain more than 2d query vectors, it can obtain 2d unknown index vectors (N a ,N b ) construct 2d formulas (17) and recover N from them a and N b However, due to the existence of the random number ε, this attack is not feasible in the present invention. ε in each query vector is different. Even if the attacker constructs 2d equations from 2d query vectors, it cannot use linear analysis to solve the equations from N a and N b 4d unknown numbers and 2d random numbers ε.

[0152] Therefore, under a known background attack model, the present invention can protect the privacy of indexes and queries.

[0153] In summary, the present invention can prevent attackers from learning indexes and query vectors under known ciphertext attack models and known background attack models.

[0154] S9-1-2. Query unlinkability.

[0155] The present invention can ensure that an attacker cannot distinguish any two queries generated from the same query image.

[0156] Proof: Assume that two feature vectors bq1 and bq2 are extracted from the query, and different ε1, ε2, β1 and β2 are randomly selected to expand bq1 and bq2 into two different bq1' and bq2'. Then, two different ξ1 and ξ2 are selected to encrypt bq1' and bq2'. The encryption formula is:

[0157]

[0158] After these operations are completed, the ciphertext E W (bq1) and E W (bq2) will become completely different. Therefore, according to E W (bq1) and E W (bq2), the opponent cannot distinguish E W (bq1) and E W (bq2) is generated from the image or text of the same query. In other words, the present invention can ensure the unlinkability of queries.

[0159] S9-2. Correctness verification.

[0160] The present invention determines the similarity by comparing the inner product of the index vector and the query vector. The calculation process is as follows:

[0161]

[0162] Since γ>>2|max(ξ)|, formula (19) can be transformed into:

[0163]

[0164] Since scaling the inner product results to the same degree does not affect the comparison of the inner product sizes, the comparison of E M (bv)E W The size of the inner product of (bq) is the size of the inner product of bv'bq'. At the same time, due to the addition of α, β∈Z when the vector is expanded p2 , so it will not affect the comparison of inner products. Formula (20) can be equivalent to:

[0165]

[0166] So calculate EM (bv) and E W The inner product of (bq) is equivalent to calculating the inner product of bv and bq. Encrypting the index vector and query vector will not affect the correctness of the retrieval.

[0167] S10. Performance evaluation

[0168] Using Flickr30K and COCO datasets, comprehensive and extensive experiments are conducted to verify the effectiveness of the present invention.

[0169] (1) Experimental environment. The method of the present invention is deployed on an Intel Core TM On a machine with an i7 CPU and a single NVIDIAGEFORCE RTX 3080Ti GPU, python 3.9 is used as the programming language.

[0170] (2) Datasets. The image datasets used in this experiment are Flickr30K and COCO. The Flickr30K dataset contains 31,783 images, each with 5 text descriptions. The present invention uses 29,783 images for training, 1,000 images for verification, and 1,000 images for testing. The COCO dataset contains 123,287 images, each with 5 corresponding text descriptions. The present invention uses 113,287 images for training, 5,000 images for verification, and 5,000 images for testing.

[0171] (3) Implementation details. The present invention analyzes various indicators of the teacher model and four student models. Teacher model T uses Faster R-CNN to extract image features, and uses a four-layer Transformer encoder to encode image features, and uses BERT to extract text features. Student model S1 uses a single-layer Transformer encoder to encode image features, and uses LSTM to extract text features. Student model S2 replaces the single-layer Transformer with a double-layer based on S1. Student model S3 replaces LSTM with Bi-LSTM based on S1. Student model S4 uses a double-layer Transformer encoder to encode features, and uses Bi-LSTM to extract text features.

[0172] When calculating the contrast loss, the edge parameter δ=0.2 and the slack variable σ=0.3. During the training process, the present invention uses Adam as the optimizer. The initial learning rate, the number of training cycles and the batch size are 1e-6, 30 and 40 respectively. During the knowledge distillation process, the balance parameter χ between the training loss and the distillation loss is adjusted to 0.2. The sensitivity of these slack variables is analyzed in detail in subsequent experiments.

[0173] Compare the model size and distillation effect when the teacher model and the student model use different network structures.

[0174] S10-1. Storage overhead evaluation.

[0175] In order to apply the model to lightweight devices, the storage overhead of the model should be compressed as much as possible.

[0176] Table 1: Storage overhead

[0177] method Storage overhead (KB) Compression S1 588616 46.71％ S2 679352 53.91％ S3 713814 56.65％ S4 762928 60.54％ T 1260138 100.00％

[0178] As shown in Table 1, in the teacher model, a four-layer Transformer encoder is used in the image feature encoding task, and a BERT model is used in text processing, with a storage overhead of 1260138KB. The present invention tests two image encoders and two text feature models. After the Transformer encoder is reduced to 1 layer and the text processing model is replaced by LSTM, the compression effect is the best, with a compression ratio of 46.71%. The two-layer Transformer encoder and Bi-LSTM have the worst compression effect, but still have 60.54%.

[0179] S10-2. Retrieval accuracy evaluation.

[0180] As shown in Table 2 and Table 3, the present invention tests the retrieval accuracy of the teacher model and the student model in the two datasets of COCO 5k and Flickr-30k. Rank at top-k (R@k) is widely used in the field of image and text retrieval as an evaluation index of retrieval accuracy. It is defined as the proportion of positive samples in the top k retrieval results. The present invention tests the retrieval accuracy when k is equal to 1, 5, and 10 through a large number of experiments.

[0181] Table 2: Comparison of retrieval accuracy between student model and teacher model under COCO dataset

[0182]

[0183] Table 3: Comparison of retrieval accuracy between student model and teacher model in Flickr30k dataset

[0184]

[0185]

[0186] Since the model is changed from a complex model to a simple model, the retrieval accuracy is reduced. The knowledge distillation technology is to minimize the loss of retrieval accuracy. Table 2 and Table 3 are the test results of 5000 images selected from the COCO dataset and Flickr30k dataset respectively.

[0187] like Figure 4 As shown in the experimental results, it can be seen that even if the model structure is simplified, it still has excellent retrieval accuracy.

[0188] S10-3. Comparison of the present invention with several latest technologies.

[0189] As shown in Table 4, the present invention is compared with several latest technologies. The teacher model T proposed in the present invention has excellent performance in both text-to-image retrieval and image-to-text retrieval. It is significantly better than the recent cross-modal retrieval scheme for images and texts, and most indicators are better than the most advanced TGDT-GL scheme. At the same time, the rsum value is improved by 3.2. For the four proposed student models, even if the structure of the model is simplified, it is better than the use of knowledge distillation technology and still better than some of the latest technologies. Student model S1, as the model with the most simplified model structure, has an rsum value that is 1.3 higher than the latest CRGN. Student model S4, as the best student model, has an rsum value that is 0.8 higher than the latest UARDA method.

[0190] Table 4: Comparison between the present invention and the prior art on the COCO dataset

[0191]

[0192]

[0193] As shown in Table 5, the present invention is compared with several state-of-the-art technologies. In the Flickr30k dataset, both the student model and the teacher model outperform several state-of-the-art solutions. The experimental results show that the teacher model is effective. At the same time, in a lightweight environment, the student model has a reasoning ability close to that of the teacher model and is also effective.

[0194] Table 5: Comparison between the present invention and the prior art on the Flickr30k dataset

[0195]

[0196]

[0197] S10-4. Ablation experiment.

[0198] The present invention will compare the impact of the value of each relaxation variable on the experimental results under the COCO5k dataset. Since the edge parameter δ is usually set to 0.2, this parameter is not evaluated.

[0199] like Figure 5 As shown in Figure 1, the present invention compares the influence of the slack variable σ on the retrieval accuracy, and compares the retrieval accuracy of the teacher model and the student model when σ=[0.1, 0.15, 0.2, ..., 0.4]. When σ<0.3, the retrieval accuracy increases, and when σ>0.3, the retrieval accuracy decreases. When σ=0.3, Figure 5 (a) and Figure 5 The retrieval accuracy in (b) all reached the highest value.

[0200] exist Figure 6 The present invention shows the comparison of retrieval accuracy between the student model and the teacher model when the balance parameter χ is equal to different values.

[0201] χ is used to adjust the ratio of the student model’s own knowledge and the knowledge learned from the teacher model. Although the teacher model has stronger reasoning ability, it does not mean that the more knowledge learned from the teacher model, the stronger the student model’s reasoning ability will be. Figure 6 This is proved by the fact that when χ = 0.2, that is, when the student model contains 20% of its own understanding of the samples and 80% of the knowledge transferred by the teacher model, the reasoning ability of the student model reaches the best effect.

[0202] In addition to the two relaxation variables, the present invention also compares the difference between using only the triple ranking loss (TRL) and combining it with the intra-model consistency contrast loss (ICC).

[0203] Table 6: Effect of different loss functions on retrieval accuracy

[0204]

[0205]

[0206] As shown in Table 6, on the basis of triple ranking loss, further increasing the consistency of intra-modal samples by using intra-modal consistency contrast loss will effectively increase the retrieval accuracy by about 2.7 to 4.9. The experimental results prove that the proposed intra-modal consistency contrast loss is effective.

[0207] S10-5. Indexing performance.

[0208] The performance of the quadtree index structure of the present invention is tested using 5000 images from the Flickr 30k dataset.

[0209] In cross-modal retrieval schemes, a common retrieval method is to compare the similarity between the query vector and all feature vectors. This will result in a huge retrieval time overhead, usually taking several minutes. The well-designed index structure of the scheme of the present invention enables the retrieval time to be measured in milliseconds. Therefore, instead of comparing the latest cross-modal retrieval scheme, the present invention compares the latest and most effective searchable encryption scheme with MIRSP, SEI and VFIRM.

[0210] like Figure 7 As shown, the retrieval times of the three comparison schemes and the scheme of the present invention are compared. Since the retrieval time of Image->Text and Text->Image is almost the same, the present invention expresses the retrieval time as the sum of the time required to execute Image->Text once and Text->Image once at k=10.

[0211] The use of the quadtree proposed by the present invention for retrieval has significant advantages. MIRSP and SEI use different methods to construct binary trees as retrieval structures. For the same feature vector, using a binary tree will traverse more nodes compared to a quadtree, resulting in a higher retrieval time overhead. This method does not construct an index structure, but reduces the amount of calculation of similarity calculation by compressing the feature vector, effectively reducing the retrieval time overhead. However, the performance is still not as good as the quadtree proposed by the present invention.

[0212] like Figure 8 As shown, although building the index will incur some time overhead, it is acceptable for cross-modal retrieval because the three compared solutions can only retrieve single-modal data.

[0213] VSE++: Faghri F, Fleet DJ, Kiros JR, et al. Vse++: Improving visual-semantic embeddings with hard negatives[J]. arXiv preprint arXiv:1707.05612, 2017.

[0214] DXR: Chen L, Xue Y, Mu Y, et al. Case-sse: Context-aware semantically extensible searchable symmetric encryption for encrypted cloud data[J]. IEEE transactions on services computing, 2022, 16(2): 1011-1022.

[0215] MPL:Wei J,Xu X,Yang Y,et al.Universal weighting metric learning forcross-modal matching[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2020:13005-13014.

[0216] GOT:Chen L,Gan Z,Cheng Y,et al.Graph optimal transport for cross-domainalignment[C] / / International Conference on Machine Learning.PMLR,2020:1542-1553.

[0217] CAMP:Wang Z,Liu X,Li H,et al.Camp:Cross-modal adaptive messagepassing fortext-image retrieval[C] / / Proceedings of the IEEE / CVF internationalconference on computer vision.2019:5764-5773.

[0218] SCAN:Lee K H,Chen X,Hua G,et al.Stacked cross attention for image-textmatching[C] / / Proceedings of the European conference on computer vision(ECCV).2018:201-216.

[0219] DSRAN:Wen K,Gu X,Cheng Q.Learning dual semantic relations with graphattention forimage-text matching[J].IEEE transactions on circuits and systemsfor video technology,2020,31(7):2866-2879.

[0220] CAMERA:Qu L,Liu M,Cao D,et al.Context-aware multi-view summarizationnetwork forimage-text matching[C] / / Proceedings of the 28th ACM InternationalConference on Multimedia.2020:1047-1055.

[0221] IMRAM:Chen H,Ding G,Liu X,et al.Imram:Iterative matching withrecurrent attentionmemory for cross-modal image-text retrieval[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and patternrecognition.2020:12655-12663.

[0222] CAAN:Zhang Q,Lei Z,Zhang Z,et al.Context-aware attention network forimage-textretrieval[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2020:3536-3545.

[0223] VSRN:Li K,Zhang Y,Li K,et al.Visual semantic reasoning for image-textmatching[C] / / Proceedings of the IEEE / CVF international conference oncomputer vision.2019:4654-4662.

[0224] VSRN++:Li K,Zhang Y,Li K,et al.Image-text embedding learning viavisual and textualsemantic reasoning[J].IEEE transactions on pattern analysisand machine intelligence,2022,45(1):641-656.

[0225] CGRN:Zhang Y,Zhou W,Wang M,et al.Deep relation embedding for cross-modalretrieval[J].IEEE Transactions on Image Processing,2020,30:617-627.

[0226] TERAN:Messina N,Amato G,Esuli A,et al.Fine-grained visual textualalignment forcross-modal retrieval using transformer encoders[J].ACMTransactions on Multimedia Computing,Communications,and Applications(TOMM),2021,17(4):1-23.

[0227] UARDA:Zhang K,Mao Z,Liu AA,et al.Unified adaptive relevancedistinguishable attentionnetwork for image-text matching[J].IEEE Transactionson Multimedia,2022,25:1320-1332.

[0228] TGDT-GL:Liu C,Zhang Y,Wang H,et al.Efficient token-guided image-textretrieval withconsistent multimodal contrastive training[J].IEEE Transactionson Image Processing,2023。

Claims

1. A lightweight and privacy-preserving cross-modal retrieval method in an Internet of Things environment, characterized in that: The steps include: S1. The data owner generates a key for encrypting a binary vector of the data to be uploaded and a key for encrypting a binary vector of the query data; and sends the key for encrypting the binary vector of the query data to the data user; S2. Training a cross-modal feature extraction model; S3. The data owner inputs the data in the dataset to be uploaded into the trained cross-modal feature extraction model to obtain data features; performs binary code conversion and expansion on the data features to generate a binary vector. The dataset to be uploaded includes the data and the identifier corresponding to the data; S4. Encrypt the binary vector according to the key used to encrypt the binary vector of the data to be uploaded, generate a quadtree index using the quadtree index structure and the encrypted binary vector, and encrypt the data to be uploaded; send the quadtree index and the encrypted data to be uploaded to the cloud server; S5. The data owner generates integrity verification auxiliary information AAI and a signature private key, and generates digital signature information corresponding to each data according to the signature private key and identifier; the data owner sends the AAI to the data user, and sends the identifier and digital signature information of each data to the cloud server; S6. The data user inputs the query data into the trained cross-modal feature extraction model in step S3 to obtain data features, converts and expands the data features into binary codes, and generates a binary vector corresponding to the query data; encrypts the binary vector corresponding to the query data according to the key used to encrypt the feature vector of the query data to generate a query vector, and sends the query vector to the cloud server; S7. The cloud server calculates the similarity between the query vector and each non-leaf node of the quadtree index from top to bottom and generates a query result; The cloud server generates integrity verification proof response information based on the query result and digital signature information, and sends the integrity verification proof response information and query result to the data user; S8. The data user verifies the query result based on the AAI and the integrity verification auxiliary information; When the query result is correct, the data user sends an encrypted data request to the cloud server, and the cloud server sends the encrypted data to be uploaded corresponding to the query result to the data user.

2. The lightweight and privacy-preserving cross-modal retrieval method in the Internet of Things environment according to claim 1, characterized in that: The specific method of training the cross-modal feature extraction model in step S2 is: In the cross-modal feature extraction model, the pre-trained Faster R-CNN is used as the teacher model for extracting image features, the Transformer is used as the student model of Faster R-CNN, the pre-trained BERT is used as the teacher model for extracting text features, and the LSTM is used as the student model of BERT. The student model uses the output of the teacher model as a reference for training, and the loss function is used to optimize the training process.

3. The lightweight and privacy-preserving cross-modal retrieval method in the Internet of Things environment according to claim 2, characterized in that: The calculation formula of the loss function is: L stu =χL+(1-χ)L dis Among them, L stu is the loss function, χ is the balance parameter, L is the contrast loss function, L dis The distillation loss function is calculated as the difference between the output of the student model and the teacher model; The calculation formula of the contrast loss function is: L=L a +L r Among them, L a is the intra-model consistency comparison loss, L r is the inter-modality consistency contrast loss; δ is the edge parameter, S(·) is the similarity function; I is the image sample, T is the text sample, v- is the hard negative sample index of the image modality, l- is the hard negative sample index of the text modality, and σ is the relaxation variable; The calculation formula of the distillation loss function is: Among them, x i is the output of the teacher model, y i is the output of the student model.

4. The lightweight and privacy-preserving cross-modal retrieval method in the Internet of Things environment according to claim 1, characterized in that: The specific method of generating the quadtree index in step S4 is: The encrypted binary vector is stored in the leaf node, and an internal node is generated for every four child nodes. The binary vector corresponding to each four child nodes generates a representative vector stored in the internal node until a root node and a binary vector stored in the root node are generated to obtain a quadtree index; when the last group is less than four nodes, a virtual node and a binary vector stored in the virtual node are filled, and the binary vector stored in the virtual node is all zero.

5. The lightweight and privacy-preserving cross-modal retrieval method in the Internet of Things environment according to claim 4, characterized in that: The specific method of generating the binary vector stored in the internal node is: If a binary vector stored in a non-leaf node is generated from a binary vector stored in a leaf node, the average value of the binary vectors corresponding to the four added leaf nodes at each position is calculated to generate an average vector, and the positions in the average vector that are greater than a predefined threshold are assigned a value of 1, and the positions that are less than the predefined threshold are assigned a value of 0, thereby generating a binary vector stored in a non-leaf node; If a binary vector stored in a non-leaf node is generated by a binary vector stored in a non-leaf node, as long as there is one 1 in the same position of the binary vectors corresponding to the four non-leaf nodes, the corresponding position of the generated binary vector is 1; if the binary vectors corresponding to the four non-leaf nodes are all 0 in the same position, the corresponding position of the generated binary vector is 0.

6. The lightweight and privacy-preserving cross-modal retrieval method in the Internet of Things environment according to claim 5, characterized in that: The specific method of generating the query result in step S7 is: The cloud server first calculates the similarity between the query vector and each non-leaf node, and selects the non-leaf nodes whose similarity is greater than the similarity threshold; then calculates the similarity between the query vector and all leaf nodes corresponding to the selected non-leaf nodes, and finally sorts them according to the similarity values, and selects the data identifiers corresponding to the first k leaf nodes as the query results.

7. The lightweight and privacy-preserving cross-modal retrieval method in an Internet of Things environment according to claim 1, characterized in that: The specific way for the data owner to generate the key for encrypting the binary vector and the key for encrypting the feature vector of the query data is as follows: S1a-1. The data owner generates system public parameters based on the security parameter λ, and the system public parameters include G1, G2, e, g, p, p pub , w and H, where G1 and G2 are multiplicative cyclic groups of order p, G2 = G1 × G1, g is a generator, and G1·p pub =g s , s is in Z p * is selected in G1, w is selected in G1, Z p1 Z p The range of values ​​after the hash function, G1 is from Z p1 taken out from S1a-2. Generate two random matrices M and M -1 , where M, M -1 ∈Z (2d)×(2d) , M -1 is the inverse matrix of M, d is the dimension of the eigenvector; generate random vectors α and β, where α={α1,α2,…,α d-1 }, β={β1,β2,…,β d-1 }, α1, α2, …, α d-1 ,β1,β2,…,β d-1 ∈Z p2 ; Generate a random integer γ, a random positive number ε, a random number θ and two random integer error vectors ξ o and u ; Calculate W = θ·M -1 ; S1a-3. The key key that will be used to encrypt the binary vector o ={M,α,γ,ξ o }Sent to the data owner, the key used to encrypt the feature vector of the query data u = {W,β,ε,γ,ξ u }Sent to data user.

8. The lightweight and privacy-preserving cross-modal retrieval method in the Internet of Things environment according to claim 7, characterized in that: The specific method for the data owner to generate integrity verification auxiliary information and signature private key is: Data owner is in Z p Uniformly select η, κ and τ from the equation; calculate the seed information ψ1 = η.κ.τ, ψ2 = η 2 .κ 2 .τ,…,ψ |M| =η |M| .κ |M| .τ, and integrity verification auxiliary information AAI = {η,κ,τ,H(ψ1),H(ψ2),…,H(ψ |M| )}, where H is the hash function and each image corresponds to a ψ i ; Generate signature private key K A , where K A =(r+s)H(ID o ), where r is the p *Selected from, ID o is the identification code of the data owner; and selects the identifier id of each data in the data set j , calculate the digital signature information corresponding to each data 9. The lightweight and privacy-preserving cross-modal retrieval method in an Internet of Things environment according to claim 8, characterized in that: The specific method for the cloud server to generate integrity verification proof response information is as follows: Cloud Server from Z p * randomly select c and calculate e=w c and Calculate the aggregate signature The cloud server generates integrity verification proof response information {μ,e,Sig}.

10. The lightweight and privacy-preserving cross-modal retrieval method in an Internet of Things environment according to claim 9, characterized in that: The specific way for data users to verify query results is as follows: S8-1. Data users calculate φ based on AAI. The calculation formula for φ is: S8-2. Using the value of φ to judge Is it true? If so, the query result is complete. If not, the query result is incomplete.

Images