Multi-Label Privacy-Preserving Image Retrieval Method Based on Object Detection in Cloud Environment

Abstract

The present invention provides a multi-label privacy protection image retrieval method based on object detection in a cloud environment. The method includes the following steps: First, use the object detection method to extract image labels. Second, use the associated label algorithm to mine frequent label sets. Then, use the vgg-16 model to extract the feature vectors of the images; next, use AES to encrypt the images and image labels, and use asymmetric scalar product preservation to encrypt the image feature vectors; and create label and feature vector indexes for each frequent label set and feature vector respectively. Finally, users with query requirements need to be authorized by the data owner first, then match the frequent label sets to narrow the retrieval range, perform similarity search based on the matched frequent label sets and feature vectors, and if no matched frequent label sets are found, search in other label sets. Compared with the existing searchable encrypted image retrieval methods, the method of the present invention not only reduces the search time, but also improves the accuracy of image retrieval.

Description

Technical Field

[0001] The present invention relates to the field of searchable encryption technology, and more particularly to a multi-label privacy protection image retrieval method based on object detection in a cloud environment. Background Art

[0002] Nowadays, cloud computing is becoming increasingly popular. More and more users outsource their image data to cloud storage in order to save their own resource costs. However, since the cloud server is not under the control of the data owner, the privacy of the data stored on the cloud server is seriously threatened. Therefore, the data owner encrypts the images to be stored and then uploads them to the cloud server. However, the cloud server cannot access the encrypted data and cannot process the user's requests. Based on this problem, Song et al. first proposed a password technology that supports searching after data encryption, opening up a new research direction in cryptography - searchable encryption. This technology can well solve the problem of encrypted image retrieval.

[0003] Hirata et al. proposed content-based image retrieval technology (CBIR). Its principle is to perform similarity search in an image database according to the visual content of the image, which is closer to human vision and helps to quickly retrieve the required images. The proposal of CBIR has attracted the attention of many scholars in this field. However, although the existing encrypted image retrieval schemes based on CBIR achieve secure and fast search, they all search in the entire encrypted database. Usually, the number of images searched by users is much smaller than the number of images stored in the database. Moreover, since an image consists of color information and texture information, two images with completely different objects but the same color and similar texture information are very likely to appear in the same search list.

[0004] With the batch increase of the database, the room for improvement in the search efficiency and accuracy of these methods is limited. To solve the above problems, while ensuring security and improving retrieval performance and accuracy, the present invention proposes a new searchable encryption image retrieval method - multi-label privacy protection image retrieval method, which can not only ensure data security but also meet the query efficiency of databases of any order of magnitude. Summary of the Invention

[0005] The object of the present invention is to provide a multi-label privacy protection image retrieval method based on object detection in a cloud environment. This method is applicable to encrypted retrieval schemes for databases of any order of magnitude, can adjust the size of the database and the extraction of image labels according to the applicable scenario, improve the retrieval efficiency, and can effectively prevent data leakage and improve the security of the privacy protection image retrieval scheme.

[0006] The present invention is implemented as follows:

[0007] A multi-label privacy protection image retrieval method based on object detection in a cloud environment, comprising the following steps:

[0008] a. Image preprocessing;

[0009] Before image encryption, it is necessary to perform preprocessing on it. According to image recognition technology, object detection is performed on each image, and multiple detected labels in the image are extracted. Using data mining technology, an association rule algorithm suitable for label detection - Association Labeling is proposed, and this algorithm is used to make an association judgment on the multiple detected labels, and the frequently occurring labels are combined to generate a frequent label set. Specifically: perform label recognition on all images. All the labels on each image are combined to form the label universe of this image, and the label universe of each image contains several label subsets. The label universes and label subsets of all images are summarized together to form a label set. For each set in the label set (including the label universe and all label subsets), it is necessary to judge whether it is a frequent label set. It is necessary to judge the proportion of the number of times this set appears in the total number of all sets in the label set. This proportion is called the support degree; when the support degree is greater than the set threshold, this set is classified as a frequent label set (the so-called frequent label set means that this set appears with a relatively high frequency); otherwise, the labels in this set are regarded as unassociated labels, and all unassociated labels are summarized together to form other label sets.

[0010] Both the frequent label set and other label sets contain not only labels but also image IDs. For a certain frequent label set, if it is the label universe or a subset of a certain image, the ID of this image is added to this frequent label set. After the image IDs in all frequent label sets are known, by taking the difference, the remaining image IDs are classified into other label sets.

[0011] For the images in each frequent label set and other label sets, use the vgg-16 model to extract feature vectors for the images in each set.

[0012] b. Encrypt various types of data after image preprocessing according to the encryption mechanism;

[0013] b-1. Encrypt images and labels;

[0014] Use the Advanced Encryption Standard (AES) to encrypt images and labels.

[0015] b-2. Encrypt image feature vectors;

[0016] The feature vectors of images are encrypted using Asymmetric Scalar-product Preserving Encryption (ASPE) to ensure a secure retrieval channel. Here, the encryption method for the feature vectors of images in the database is different from that for the feature vectors of the images to be retrieved during user retrieval, but both use ASPE encryption.

[0017] c. Create indexes;

[0018] Construct encrypted indexes for the encrypted tags and feature vectors respectively to form a tag index and a feature vector index.

[0019] d. Encrypt the preprocessed image data respectively and then upload them to the cloud server.

[0020] e. Request retrieval;

[0021] Users with retrieval requirements send retrieval requests to the cloud server after being authorized by the image owner. The cloud server matches the frequent tag sets to narrow the retrieval scope, and performs similarity searches based on the matched frequent tag sets and feature vectors. If no matched frequent tag sets are found, it directly searches in other tag sets.

[0022] f. After step e is executed, the cloud server returns the retrieval results to the user, and the user decrypts the images using the decryption key to obtain the plaintext query results.

[0023] In the above solution, the specific processes for generating the encrypted tag index and the encrypted feature vector index are as follows:

[0024] Traverse the images in each frequent tag set and other tag sets, and extract all the image feature vectors. Use the ASPE encryption algorithm to encrypt the extracted feature vectors one by one, and add the encrypted feature vectors to the encrypted image feature vector set to construct the encrypted feature vector index. During the process of obtaining the frequent tag sets, the tags that are not assigned to the frequent tag sets are automatically added to other tag sets because they do not meet the support degree. Both other tag sets and each frequent tag set store not only the tags but also the image ids corresponding to their tags. When creating a frequent tag set, if the frequent tag set is the full set of tags or a subset of a certain image, obtain the id of this image and add it to this frequent tag set. Encrypt all the tags using the encryption algorithm of the encrypted images. Create a frequent tag index using all the frequent tag sets, and add other tag sets to this index to construct a complete tag index. After the above processes are completed, the encrypted feature vector index and the encrypted tag index are generated.

[0025] In step e, the specific process of the retrieval stage is as follows:

[0026] The retrieval stage involves two steps: ① Generate a trapdoor: TradDoor; ② Trapdoor retrieval: Search.

[0027] The following elaborates on each step.

[0028] TradDoor: When a user wants to search for some images from the image database stored in the cloud server, the user needs to first obtain authorization from the corresponding data owner of the database. After authentication: First, the user extracts multiple tags L{l1, l2, l3.....} representing the image content and a d-dimensional image feature vector f from the query image. q 。

[0029]

[0030] Secondly, expand the feature vector. Expand f q from d dimensions to d + 1 dimensions according to formula (2):

[0031] f q = r * (f q T , 1) T (2)

[0032] r is a random number greater than 0. The first d dimensions are the original f q , and the (d + 1)-th dimension is 1.

[0033] Thirdly, the user encrypts the feature vector and the extracted multiple tags. Encrypt the feature vector according to formula (3): where M is a random matrix.

[0034] f Eq = M -1 * f q (3)

[0035] Finally, the user sends the processed search trapdoor to the cloud server for subsequent search operations.

[0036] Search: The cloud server runs this search algorithm and returns the retrieved similar results to the requesting user. The cloud server first matches the trapdoor with the frequent tag sets in the database. Then, look for the frequent tag sets included in the entire set of query image tags. If there are matching frequent tag sets, read the set and obtain the image ids in the set. Finally, in the feature vector index, perform a secure KNN calculation based on the feature vectors corresponding to the above image ids. According to formula (4), judge the Euclidean distance between each image feature vector and the query image feature vector, obtain the top-k results, and return the results to the requesting user for the next step of operation. If no frequent tag sets are matched, directly search in other tag sets.

[0037]

[0038] Since r is a random number greater than 0, the inequality 0.5r(d(f v2 , f q ) - d(f v1 , f q )) > 0 can be used to correctly obtain d(f v2 , f q ) > d(f v1 , f q ).

[0039] The present invention introduces image recognition technology to design a multi-label privacy-preserving image retrieval method. The target detection method is used to achieve fine-grained image classification. This process only extracts image content labels in a targeted manner, effectively narrowing the retrieval range, thereby improving the query and retrieval performance of searchable encryption during the retrieval process. Based on data mining technology, a two-layer index is designed using label association rules. Through the first-layer index (label index), the images are correctly classified and the search range is determined, and within the delimited search range, the search results are determined through the second-layer index (feature vector index). Retrieving within the same category improves the search accuracy. At the same time, according to the nature of the data itself, two encryption methods are adopted, that is, AES encryption is used for images and labels, and ASPE encryption is used for feature vectors, further enhancing the security of encrypted image retrieval.

[0040] The present invention abandons the traditional privacy-preserving image retrieval method of directly creating a vector index based on content, and relies on image recognition to extract image labels to narrow the database retrieval range and accurately locate the retrieval target. The label association rules perform correlation clustering on multiple labels of image processing to create a label index. In order to search for images more accurately, the cloud server calculates the Euclidean distance of image feature vectors to calculate the similarity score for the multi-label image set and generates the top-k most relevant query results. The present invention combines the methods of searchable encryption and image recognition, which not only ensures the security of image data, but also reduces the retrieval time by about 6 times and improves the retrieval accuracy by about 15%. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is the flowchart of the method of the present invention.

[0042] Figure 2 is the query example diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] As Figure 1As shown in the figure, the multi-label privacy protection image retrieval method based on object detection in the cloud computing environment provided by the present invention starts from the perspective of artificial intelligence image object detection, and uses multi-label combination classification to preprocess the image, thereby narrowing the retrieval range. Different from most encrypted image retrievals, it does not retrieve in the entire image database or the entire encrypted domain, but retrieves within the range of the matching frequent label set, greatly improving the retrieval time. To ensure the privacy of different data types, different encryption methods are used for images, labels, and feature vectors; to facilitate retrieval, after matching the frequent label set, the similarity score is directly calculated based on the feature vector corresponding to the image id within it; to represent the image more accurately, the cloud server generates the query result with the strongest similarity by calculating the Euclidean distance.

[0044] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0045] The data owner preprocesses the images in the database and uploads them to the cloud server, which specifically includes the following steps:

[0046] Step 1: Identify each target label included in the image according to YOLOv5.

[0047] Step 2: Traverse the labels of each image according to the label association rule algorithm designed by data mining, and determine the frequent label set and other label sets according to the image labels. In order to ensure the retrieval accuracy of the obtained frequent label set without consuming too much retrieval time, the support degree of the frequent label set needs to be reasonably set. The present invention verifies that the support degree threshold of the label association rule is not less than 0.05. In a specific embodiment, the threshold can be selected as 0.05, that is: when the ratio of a certain set in the label set to the total number of all sets in the label set is greater than 0.05, this set is classified as a frequent label set.

[0048] Determine the frequent label set according to the id of the image and the support degree of each set in the label set, and automatically group the other unassociated labels except the frequent label set to form other label sets.

[0049] Step 3: Use the vgg-16 model in the convolutional neural network to extract the image feature vector, encrypt the image and label with AES, and encrypt the feature vector with the asymmetric scalar product ASPE. When using ASPE encryption, first expand the feature vector to increase its dimension by 1, and then encrypt the expanded feature vector using the following encryption method:

[0050] f Ev = M T * f v where f v represents the expanded feature vector of the image in the database, fEv Represents the encrypted form of the image feature vector in the database.

[0051] Step Four: Use the encrypted tags and feature vectors to construct encrypted indexes respectively to form a two-layer index for retrieval.

[0052] Step Five: Upload all the above encrypted data to the cloud server for storage and retrieval by users.

[0053] The user's retrieval stage includes the following steps:

[0054] Step One: The user with retrieval requirements first obtains authorization from the data owner.

[0055] Step Two: The user extracts the full set of tags of the requested query image through object detection and extracts its feature vector through vgg-16.

[0056] Step Three: Encrypt the requested query image, the full set of tags of the query image, and the feature vector to generate a request trapdoor.

[0057] Use AES to encrypt the image and the full set of tags; expand the feature vector and use ASPE to encrypt the expanded feature vector. The method of encrypting the feature vector of the query image in this step is different from the method of encrypting the feature vector of the image in the database above. The method of encrypting the feature vector of the query image in this step is as follows:

[0058] f Eq = M -1 * f q where f q represents the expanded feature vector of the query image, and f Eq represents the encrypted form of the feature vector of the query image.

[0059] Applying different encryption methods to process the feature vectors of the images in the database and the query image can effectively ensure the privacy of the outsourced data.

[0060] Step Four: The user uploads the request trapdoor to the cloud server.

[0061] Step 5: After the cloud server receives the data uploaded in Step 4, it first traverses the frequent tag sets pre-stored on the cloud, and then determines the frequent tag sets to be queried based on the complete tag set in the request trapdoor. If a certain frequent tag set is a subset of the complete image tag set, it is considered a successful match, and the image is searched for in that frequent tag set (although the frequent tag set is a subset, all image IDs containing this tag exist in this frequent tag set); it is also possible to match multiple frequent tag sets; finally, the images in the frequent tag sets are filtered through KNN to obtain similar results; if there are multiple successfully matched frequent tag sets, the Euclidean distance is calculated in each frequent tag set, and then all the Euclidean distances are sorted. Here, if no suitable frequent tag set is matched, similarity retrieval is automatically performed in other tag sets.

[0062] Step 6: After obtaining the similar search results, the cloud server returns them to the user for decryption and browsing.

[0063] The present invention will be described in detail below with reference to actual examples. The example assumes searching for the top-9 similar results and using the Caltech256 dataset as the real database.

[0064] 1. Identify the image tags, extract the feature vectors, and construct an encrypted index.

[0065] Step 1: Traverse the database DataSet = {Img1, Img2, ……, Img n}(n represents the number of images in the database) in the given Caltech 256 dataset, extract the tags of the images to form an image tag set imgLabelSet. This image tag set contains all the tags extracted from all the images in the database DataSet and is arranged in the form of a set.

[0066] Step 2: Filter the frequently occurring tag sets according to the support degree of the association tag algorithm (the support degree threshold is set to 0.05 here) and generate the frequent tag set frequent_label_groups.

[0067] Step 3: Scan and traverse the entire tag set.

[0068] Step 4: Find all the frequent tag sets in the tag set that meet the support degree.

[0069] Step 5: Extract the feature vectors of the images in each frequent tag set.

[0070] Step 6: Use the Advanced Encryption Standard (AES) to encrypt each frequent tag set and the images, and encrypt the corresponding feature vectors according to ASPE.

[0071] Step 7: Construct an encrypted index to ensure retrieval security.

[0072] II. Retrieve based on the encrypted data stored in Step 1, as Figure 2 shown.

[0073] Step 1: Provide a request image of the user.

[0074] Step 2: Obtain the complete set of labels and feature vectors of the image respectively according to the operation sequence of extracting feature vectors and constructing an encrypted index by identifying image labels.

[0075] Step 3: Extract the results of the above operations. The results of the above operations show that the complete set of labels included in the image is forzenset = ({0, 34}). Here, according to the labels represented by the numbers in the image label set imgLabelSet, where 0 represents ['person'] and 34 represents ['baseball bat'].

[0076] Step 4: Encrypt the complete set of labels forzenset = ({0, 34}) and the extracted image vector in Step 3 above to form a query trapdoor.

[0077] Step 5: The querying user sends a trapdoor request to the cloud server.

[0078] Step 6: After receiving the query request, the cloud server matches the encrypted complete set of labels in the trapdoor with the pre-stored frequent label sets to determine which one or which frequent label sets in the cloud server the complete set of labels of this image belongs to.

[0079] Step 7: After determining the matching frequent label set, perform KNN calculation according to the encrypted vector in the corresponding frequent label set, and obtain the similarity result according to the Euclidean distance. If no appropriate frequent label set can be matched, directly perform similarity calculation in other label sets.

[0080] Step 8: The cloud server returns the top-9 similarity results processed in Step 7 above to the user.

[0081] Step 9: After receiving the retrieval result, the user decrypts and views it with the image decryption key authorized by the data owner.

[0082] The content not described in detail in this specification belongs to the prior art well-known to those skilled in the art.

[0083] The specific embodiments of the present invention are described above in combination with the accompanying drawings and technical solutions. The above embodiments are only preferred embodiments given to fully illustrate the present invention. The protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A multi-label privacy-preserving image retrieval method based on object detection in a cloud environment, characterized in that The steps include: a. Image preprocessing; According to the image recognition technology, each image in the database is detected and the label is extracted. The label association rule algorithm is used to frequently combine the above labels to obtain a frequent label set. Specifically, all images are labeled, and the full set of labels and label subsets of all images are aggregated together to form a label set. To determine whether each set in the label set is a frequent label set, it is necessary to determine the proportion of the number of times the set appears to the number of all sets in the label set, and this proportion is called support. If the support is greater than the set threshold, the set is classified as a frequent label set. Otherwise, the labels in the set are regarded as unrelated labels, and all unrelated labels are aggregated together to form other label sets. Frequent tag sets and other tag sets contain not only tags but also image IDs. For a frequent tag set, if it is the full set of tags for a certain image or a subset of them, the ID of the image is added to the frequent tag set. After all the image IDs in the frequent label sets are known, the remaining image IDs are classified into other label sets by subtraction; For each frequent label set and images in other label sets, the vgg-16 model is used to extract feature vectors for the images in each set; b. Encrypt various types of data after image preprocessing according to the encryption mechanism and generate indexes; b-1, encrypted images and labels; Use AES to encrypt images and image tags; b-2, encrypted image feature vector; The feature vector of the encrypted image is encrypted using an asymmetric scalar preserving product; b-3. Construct encrypted indexes for the encrypted labels and feature vectors respectively; c. Encrypt the pre-processed image data and upload them to the cloud server; d. request for retrieval; Users with search requirements send search requests to the cloud server after authorization by the image owner. The cloud server matches frequent tag sets to narrow the search scope and performs similarity search based on the matched frequent tag sets. If there is no matched frequent tag set, it directly searches in other tag sets. e. After step d is executed, the cloud server returns the search results to the user, and the user decrypts the image using the decryption key to obtain the plaintext query results.

2. The multi-label privacy-preserving image retrieval method based on object detection in a cloud environment according to claim 1, wherein In step b-2, an asymmetric scalar preserving product is used to encrypt the feature vector of the image, as follows: First, the feature vector is expanded to increase its dimension by 1, and then the expanded feature vector is encrypted using the following encryption method: f Ev = M T * f v , where f v represents the feature vector after the expansion of the image in the database, and f Ev represents the encrypted form of the feature vector of the image in the database; M is a random matrix.

3. The multi-label privacy protection image retrieval method based on object detection in a cloud environment according to claim 1, characterized in that, In step d, the user first generates a trapdoor based on the image to be queried, and then makes a retrieval request; The method for generating a trapdoor is as follows: First, the user extracts an image label and a feature vector f from the query image q ; Second, the feature vector is extended to increase the dimension of the feature vector by 1; Then, the label is encrypted using AES, and the extended feature vector is encrypted using the following encryption method: f Eq = M -1 * f q , where f q represents the feature vector of the query image after expansion, and f Eq represents the encrypted form of the feature vector of the query image; M is a random matrix.

4. The multi-label privacy-preserving image retrieval method based on object detection in a cloud environment according to claim 3, characterized in that In step d, when the cloud server matches the frequent tag set, if a certain frequent tag set is a subset of the query image tag set, it means that the match is successful; then find all the image IDs in the frequent tag set, and then find the feature vector corresponding to the image ID according to the index, and use KNN calculation to determine the Euclidean distance between each image feature vector and the query image feature vector according to formula (4), get the top-k result, and return the result to the requesting user, who will proceed to the next step; if there is no matching frequent tag set, search directly in other tag sets; Since r is a random number greater than 0, the inequality 0.5r(d(f v2 ,f q ) - d(f v1 ,f q )) > 0 can be used to correctly obtain d(f v2 ,f q ) > d(f v1 ,f q ).

Images