Cross-modal image-text retrieval method for privacy protection of Chinese users

Abstract

The invention provides a Chinese user-oriented privacy protection cross-modal image-text retrieval method, which comprises the following steps of: submitting image-text data to a private server by a user, mapping image-text features to the same semantic space through a feature extraction model designed by an efficient parameter fine tuning method, and encrypting by adopting an asymmetric scalar product technology; uploading the encrypted image-text data and the image-text data features to a cloud server, and constructing a hierarchical graph index of the encrypted data by the cloud server; when a user queries, the private server extracts and encrypts query features, the cloud server executes approximate nearest neighbor search through the established index structure, an encryption result is returned to the private server for decryption, and finally a plaintext result is sent to the user. According to the method and the device, the cross-modal retrieval service for the image-text data stored in the third-party cloud server can be provided for the user while the privacy security of the user data is protected, and the method and the device have relatively high retrieval precision and retrieval efficiency and relatively low training overhead.

Description

Technical Field

[0001] The present application relates to the field of cross-modal retrieval, and in particular to a cross-modal image and text retrieval method with privacy protection for Chinese users. Background Art

[0002] With the rapid development of the Internet and information technology, personal multimedia data, mainly pictures and texts, has exploded, and with it comes an increasing demand for data storage space. Therefore, more and more users tend to store their private multimedia data in third-party cloud servers to save local storage space, which brings with it the issue of user data privacy and security. How to effectively protect user data privacy while providing users with accurate and fast cross-modal data retrieval services has become a hot issue that needs to be studied in the field of privacy protection.

[0003] In addition, existing privacy-preserving cross-modal retrieval solutions are designed and tested based on English datasets, which limits the use of Chinese community users. Although the data of Chinese users can be translated into English before storage and retrieval, this results in a waste of computing resources, and semantic distortion and errors are inevitable under machine translation. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] This application proposes a privacy-preserving cross-modal image-text retrieval method for Chinese users.

[0006] According to a first aspect of an embodiment of the present application, a cross-modal image and text retrieval method with privacy protection for Chinese users is provided, comprising:

[0007] In response to receiving the graphic data sent by the user, extracting features from the graphic data using the feature extraction model deployed thereon, and embedding and mapping the features of the extracted graphic data into the same semantic space;

[0008] Encrypt the text and pictures in the user's graphic data using text encryption technology and image encryption technology respectively, and encrypt the features of the extracted graphic data using asymmetric scalar product preservation encryption technology, and send the encrypted graphic data and the encrypted graphic data feature set to the cloud server;

[0009] In response to a query request initiated by a user, feature extraction and encryption are performed on the query content input by the user to obtain an encrypted query trapdoor and send it to the cloud server;

[0010] In response to receiving the encrypted query result returned by the cloud server, the encrypted query result is decrypted and the decrypted plaintext query result is returned to the user.

[0011] Optionally, also include:

[0012] The feature extraction model is based on the Chinese multimodal pre-trained large model CN-CLIP with all original weight parameters frozen, and is constructed by adding a General Adapter module in front of the text end of the original basic model. The General Adapter model includes a set of learnable vector pseudo prompts and a model component Prompt Encoder that can map the learnable vector pseudo prompts to a specific semantic space.

[0013] Optionally, the training process of the feature extraction model includes:

[0014] For the image-text retrieval task and the zero-shot classification task of an image, the training methods of contrastive learning and multi-label learning are combined respectively, and a large-scale Chinese image-text dataset in a general field is used to train the added General Adapter module.

[0015] Optionally, the step of encrypting the features of the extracted graphic data using an asymmetric scalar product preserving encryption technique further includes:

[0016] Through a random key generation algorithm, keys M1, M2 and S are obtained, wherein M1 and M2 are random reversible matrices of dimension (n+3)×(n+3), S is a random vector of dimension 1×(n+2), and the value of the random vector S is composed of randomly distributed 0s and 1s, wherein all keys are open to access and use rights for the private server;

[0017] For the image and text data feature E = {e1, e2, ..., e m} in the i-th feature e i , generate a random number ε i , e i becomes

[0018] Use key S to i Decompose into e′ i and e″ i , the principle of decomposition is: if S[j] = 0, then e′ i [j]=e″ i [j], otherwise, e′ i [j]+e″ i [j] = e i [j];

[0019] Use keys M1 and M2 to encrypt e′ i and e″ i Encryption, feature e i is encrypted as {e′ i M1,e″ i M2}, denoted as

[0020] Optionally, in response to a query request initiated by a user, extracting features and encrypting the query content input by the user, obtaining an encrypted query trapdoor and sending it to a cloud server, includes:

[0021] In response to the query request initiated by the user, feature extraction is performed on the query content input by the user, and the feature embedding E Q , embed the features of user queries into E according to random numbers Q becomes The expression is:

[0022]

[0023] in, is the transformed feature embedding, r and t are random numbers generated by the private server;

[0024] Use key S to Decompose into E′ Q and E″ Q , the decomposition principle is: for j∈[1,n+2], if S[j]=1, then E′ Q [j]=E″ Q [j]; otherwise, it is E′ Q [j]+E″ Q [j] = E Q [j];

[0025] Use the keys separately and For E′ Q and E″ Q Encryption is performed to obtain the encrypted query trapdoor, the expression is:

[0026]

[0027] in, A trapdoor for encrypted queries.

[0028] According to a second aspect of an embodiment of the present application, a cross-modal image and text retrieval method with privacy protection for Chinese users is provided, comprising:

[0029] In response to receiving the encrypted graphic data input by the user and the encrypted graphic data feature set sent by the private server, constructing a hierarchical graph-based index for the encrypted data feature set using the retrieval model deployed thereon;

[0030] In response to receiving the encrypted query trapdoor sent by the private server, performing an approximate nearest neighbor search for the query trapdoor in the established index structure to obtain the most appropriate encrypted query result;

[0031] The most suitable encrypted query result is returned to the private server.

[0032] Optionally, the step of constructing a hierarchical graph-based index on the encrypted data feature set using the retrieval model deployed thereon includes:

[0033] Create an empty graph structure, treat each encrypted data feature embedding as a vector node, and insert nodes one by one. Each new node will be randomly assigned a level. The higher the level, the fewer nodes there are.

[0034] When inserting a new node, start from the highest level and use greedy search to find the nearest neighbor of the level to which the current node belongs, and repeat this process at lower levels to determine more precise neighbor nodes;

[0035] After finding the nearest neighbor nodes, the inserted new node is connected to these neighbor nodes and the graph structure is updated.

[0036] Optionally, in response to receiving the encrypted query trapdoor sent by the private server, performing an approximate nearest neighbor search for the query trapdoor in the established index structure to obtain the most suitable encrypted query result includes:

[0037] When searching, start from the entry node of the highest level in the hierarchical graph and use greedy search to find the node in the current level that is closest to the query vector;

[0038] After finding the node closest to the query vector in the current layer, search downward layer by layer, repeating this process at each layer to keep getting closer to the query vector;

[0039] After reaching the bottom layer, an exact search is performed to find the node closest to the query vector.

[0040] Optionally, also include:

[0041] During the search process, the cosine distance is used as the distance between the query trapdoor and the vector node on the index structure, and the calculation expression is:

[0042]

[0043] Wherein, L is the cosine distance, and the inverse of the cosine distance represents the cosine similarity between the encrypted query trapdoor and the ciphertext data corresponding to the vector node on the index structure;

[0044] The ciphertext data corresponding to the vector node closest to the encrypted query trapdoor is taken as the most suitable encrypted query result.

[0045] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0046] This application improves the model's feature extraction capability for Chinese graphic data by adding a General Adapter consisting of a set of learnable vectors and a trainable encoder for mapping these vectors before the text end of the Chinese pre-trained model, and combines contrastive learning and multi-label learning methods for training. In the whole process, only a small number of parameters in the General Adapter need to be trained; by using asymmetric scalar product preserving encryption technology, the privacy security of users during data storage and retrieval operations on cloud servers is protected; by using the HNSW vector approximate nearest neighbor search algorithm, the efficiency of the retrieval model when matching and searching for query trapdoors on cloud servers is improved. Under the premise of protecting the privacy security of user data, this application has strong cross-modal retrieval and zero-sample classification capabilities for Chinese graphic data sets, as well as low training overhead. This application can be used for cross-modal retrieval of Chinese graphic data on outsourced servers.

[0047] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 is a system framework diagram of a privacy-protected cross-modal image-text retrieval method for Chinese users according to an exemplary embodiment;

[0050] Figure 2 is a flowchart of a privacy-protected cross-modal image and text retrieval method for Chinese users according to an exemplary embodiment;

[0051] Figure 3 The present invention is a training process of a feature extraction model in a picture and text retrieval task according to an exemplary embodiment;

[0052] Figure 4 This is a training process of a feature extraction model in a zero-shot classification task of an image according to an exemplary embodiment;

[0053] Figure 5 is a flowchart of a privacy-protected cross-modal image and text retrieval method for Chinese users according to an exemplary embodiment;

[0054] Figure 6 is a schematic diagram showing a comparison of recall rates of various models when performing a text search task based on images on CN-Flickr30K according to an exemplary embodiment;

[0055] Figure 7 is a schematic diagram showing a comparison of recall rates of various models when performing a text-to-image search task on CN-COCO according to an exemplary embodiment;

[0056] Figure 8 is a schematic diagram showing the accuracy of each model when performing a zero-sample classification task of an image on a Chinese image classification dataset according to an exemplary embodiment;

[0057] Fig. 9 is a schematic diagram showing how the time consumption of a feature extraction model in generating encrypted feature embedding vectors of files varies with the number of files according to an exemplary embodiment;

[0058] Fig.10 is a schematic diagram showing how the time consumption of a feature extraction model when constructing an index using encrypted feature embedding vectors of files varies with the number of files according to an exemplary embodiment;

[0059] Fig.11 It is a schematic diagram showing how the time consumed by different models for searching for a user query vector on a built index structure varies with the number of files according to an exemplary embodiment. DETAILED DESCRIPTION

[0060] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0062] This application proposes a privacy-preserving cross-modal image and text retrieval method for Chinese users, aiming to solve the problem that Chinese community users have difficulty in performing efficient and accurate cross-modal retrieval of outsourced image and text data on third-party cloud servers, as well as the privacy and security issues in the process of storing and retrieving data.

[0063] Figure 1 It is a system framework diagram of a privacy-preserving cross-modal image-text retrieval method for Chinese users according to an exemplary embodiment, which describes the interaction process between different entities.

[0064] Figure 2 The flowchart of a privacy-protected cross-modal image and text retrieval method for Chinese users according to an exemplary embodiment includes the following steps:

[0065] Step 101, in response to receiving the graphic data sent by the user, the feature extraction model deployed thereon is used to extract features from the graphic data, and the features of the extracted graphic data are embedded and mapped in the same semantic space.

[0066] This step is the processing process on a private server.

[0067] Efficient parameter fine-tuning technology is an efficient parameter fine-tuning technology for pre-trained large models. Since the parameters of existing pre-trained large models are very large, retraining the model using the traditional full-scale fine-tuning method is no longer a suitable option. Efficient parameter fine-tuning technology usually freezes most or all of the parameters of the original model, and adds corresponding structures for fine-tuning according to specific tasks. When training the model, only the weight parameters in the newly added structure need to be updated, which greatly reduces the computing power required for fine-tuning training. This application utilizes efficient parameter fine-tuning technology to improve the original model's feature extraction capabilities for general and professional field graphic data while introducing a small number of trainable modules.

[0068] In one embodiment of the present application, the feature extraction model deployed on the private server is based on the original multimodal pre-trained model CN-CLIP, all its original weight parameters are frozen, and a new trainable module is added at a specific position to extract the task-related embedding vector E={e1,e2,…,e m}(Assume that there are m pieces of image and text data in total, and the dimension of data feature embedding is n).

[0069] Among them, the new trainable module added at a specific position is the General Adapter module added before the text end of the original basic model. The General Adapter model includes a set of learnable vector pseudo prompts Pseudoprompts and a model component Prompt Encoder that can map the learnable vector Pseudo prompts to a specific semantic space.

[0070] It should be noted that in the vector space, Pseudo prompts refer to a set of randomly initialized vectors used to guide and prompt the model to perform specific downstream tasks. Prompt Encoder is a component composed of an MLP network or an LSTM network, which is mainly used to encode and convert Pseudo prompts into representations suitable for model input.

[0071] In addition, in one embodiment of the present application, the training process of the feature extraction model includes:

[0072] After adding the General Adapter module to the original pre-trained model, for the image-text retrieval task and the zero-shot classification task of the image, the training methods of contrastive learning and multi-label learning are combined, and the added General Adapter module is trained using a large-scale Chinese image-text dataset in a general field. After the training is completed, the model can be used to extract features from Chinese image-text data in different tasks.

[0073] Figure 3 This is a training process of a feature extraction model in a graphic and text retrieval task according to an exemplary embodiment.

[0074] like Figure 3 As shown in the figure, for the training of image-text retrieval tasks: h continuously differentiable Pseudo prompts ([P0], [P1]... [P h ]) is sent to the Prompt Encoder, and after mapping, the template embedding prompts ([V0], [V1]... [V h])). At the same time, the Chinese subtitles of the picture are sent to the Token Mapping module, which will perform operations such as word segmentation, adding special words, word embedding mapping, adding paragraph encoding and position encoding on the text tags in turn, and finally obtain the semantic feature embeddings of the picture captions ([W0], [W1]... [W s ]). After that, prompts and word embeddings are concatenated and fed into the Text Encoder to get the normalized global semantic feature embeddings of the whole sentence. Finally, the inner product is calculated with the global semantic feature embeddings of the image generated by the Image Encoder to get the cosine similarity between the image caption and the image.

[0075] In one embodiment of the present application, a large-scale Chinese text and image data set is used to train the added General Adapter module in combination with a contrastive learning training method.

[0076] Figure 4 The present invention is a training process of a feature extraction model in a zero-shot classification task of an image according to an exemplary embodiment.

[0077] like Figure 4 As shown in the figure, the training of the zero-shot classification task for images is similar to the training process of the image-text retrieval task. The original image captions are replaced with the text labels of the images and mapped to word embeddings. Then, they are combined with the prompts generated for the task and sent to the Text Encoder to be mapped to sentence embeddings. In this process, the embedding vector of each image is similar to the embedding vectors of all text labels, and the training method is multi-label learning.

[0078] Step 102, encrypt the text and images in the user's graphic data using text encryption technology and image encryption technology respectively, and encrypt the extracted features of the graphic data using asymmetric scalar product preserving encryption technology, and send the encrypted graphic data and the encrypted graphic data feature set to the cloud server.

[0079] In one embodiment of the present application, the features of the extracted graphic data are encrypted using an asymmetric scalar product preserving encryption technique.

[0080] Asymmetric Scalar Product Preserving Encryption (ASPPE) is an encryption technology designed to ensure data privacy and security. This technology is mainly used to allow scalar product calculations to be performed on encrypted data without exposing the original data. It is mainly used to retrieve outsourced data on third-party untrusted cloud servers. The present invention uses this technology to provide privacy and security protection for users to store and retrieve outsourced data.

[0081] First, the user key is generated. Through the random key generation algorithm, the keys M1, M2 and S are obtained, where M1 and M2 are random reversible matrices with dimensions (n+3)×(n+3), S is a random vector with dimensions 1×(n+2), and the value of the random vector S is composed of randomly distributed 0s and 1s. All keys are open to private servers for access and use.

[0082] Then, for the image and text data feature E = {e1, e2, ..., e m} in the i-th feature e i , generate a random number ε i , e i becomes and use the key S to i Decompose into e′ i and e″ i , the principle of decomposition is: if S[j] = 0, then e′ i [j]=e″ i [j], otherwise, e′ i [j]+e″ i [j] = e i [j]; Then, use keys M1 and M2 to encrypt e′ i and e″ i Encryption, feature e i is encrypted as {e′ i M1,e″ i M2}, denoted as

[0083] In addition, the private server uses encryption algorithms for images and texts to encrypt the user's image and text data, and then packages the encrypted image and text data together with the encrypted image and text data feature set and sends them to the cloud server.

[0084] Step 103, in response to a query request initiated by the user, feature extraction and encryption are performed on the query content input by the user to obtain an encrypted query trapdoor and send it to the cloud server.

[0085] In the embodiment of the present application, when a user wants to query, the feature embedding E of the user query is obtained. Q , the private server generates random numbers r and t, and embeds the features of the user query into E Q becomes The expression is:

[0086]

[0087] in, is the transformed feature embedding, r and t are random numbers generated by the private server.

[0088] Then, using the key S Decompose into E′ Q and E″ Q , the decomposition principle is: for j∈[1,n+2], if S[j]=1, then E′ Q [j]=E″ Q [j]; otherwise, it is E′ Q [j]+E″ Q [j] = E Q [j](The decomposition ratio is random).

[0089] Then, use the keys separately and For E′ Q and E″ Q After encryption, the query trap is finally formed as follows:

[0090]

[0091] in, A trapdoor for encrypted queries.

[0092] Step 104, in response to receiving the encrypted query result returned by the cloud server, decrypt the encrypted query result and return the decrypted plaintext query result to the user.

[0093] When the private server receives the encrypted query result returned by the cloud server, the private server decrypts the encrypted query result according to the above key, and then returns the decrypted plaintext query result to the user.

[0094] Figure 5 The flowchart of a privacy-protected cross-modal image and text retrieval method for Chinese users according to an exemplary embodiment includes the following steps:

[0095] Step 201, in response to receiving the encrypted graphic data and the encrypted graphic data feature set sent by the private server, constructing a hierarchical graph-based index for the encrypted data feature set using the retrieval model deployed thereon.

[0096] This step is the processing process on the cloud server.

[0097] In the embodiment of the present application, the cloud server regards each encrypted data feature as a vector node, and then uses the HNSW algorithm to index and construct the encrypted data features.

[0098] The HNSW (Hierarchical Navigable Small World) algorithm is an efficient vector approximate nearest neighbor search algorithm, which is mainly used to search for the nearest neighbor point in high-dimensional space, especially on large-scale data sets. The algorithm is based on the theory of navigable small world algorithm, and quickly finds the nodes that are the nearest neighbors of the query node by constructing and searching a hierarchical graph structure. This application uses this algorithm to achieve efficient search in massive high-dimensional vectors.

[0099] Specifically, the process of indexing encrypted data features using the HNSW algorithm includes:

[0100] First, an empty graph structure is created, and each encrypted data feature is embedded as a vector node. Then, nodes are inserted one by one. Each new node is randomly assigned a level. The higher the level, the fewer nodes there are. When inserting a new node, start from the highest level and use greedy search to find the nearest neighbor of the level to which the current node belongs. Repeat this process at lower levels to determine more precise neighbor nodes. After finding the nearest neighbor nodes, connect the new node to these neighbor nodes and update the graph structure. Finally, the navigability of the hierarchical graph is further optimized by adding connections and adjusting edges.

[0101] Step 202, in response to receiving the encrypted query trapdoor sent by the private server, an approximate nearest neighbor search is performed for the query trapdoor in the established index structure to obtain the most suitable encrypted query result.

[0102] In an embodiment of the present application, when a user wants to query, the private server will send an encrypted query trap to the cloud server based on the content of the user's query. The cloud server will perform an approximate nearest neighbor search for the query trap in the established index structure to obtain the most suitable encrypted query result.

[0103] When searching, we start from the entry node of the highest layer in the hierarchical graph and use greedy search to find the node in the current layer that is closest to the query vector. Then we search downward layer by layer, repeating this process at each layer to get closer to the query vector. After reaching the bottom layer, we use precise search to find the node closest to the query vector.

[0104] In addition, during the search process, the distance between the query trapdoor and the vector node on the index structure is calculated as the cosine distance, and its reciprocal represents the cosine similarity between the query and the data file corresponding to the node. The calculation process is:

[0105]

[0106] Where L is the cosine distance.

[0107] In addition, it is necessary to select appropriate random numbers r, ε i , t, in order to protect user privacy and security while minimizing its impact on retrieval accuracy.

[0108] It should be noted that the last dimension of the encrypted data feature embedding vector records the file number of the corresponding ciphertext data. Based on this, the cloud server returns the ciphertext data corresponding to the vector node closest to the query trapdoor to the private server as the most suitable encrypted query result.

[0109] Step 203: Return the most appropriate encrypted query result to the private server.

[0110] When the private server receives the most consistent encrypted query result returned by the cloud server, the private server decrypts the ciphertext data and returns the plaintext result to the user.

[0111] In one embodiment of the present application, in order to verify the effectiveness of the present application, the present application tests the recall rate comparison between the feature extraction model of the present solution and other similar models in the cross-modal image and text retrieval process, as well as the time change of the retrieval model of the present solution during the index construction and search process. The hardware environment of the experiment is: GPU: two NVIDIAGeForce RTX 4090D; CPU: 30 vCPU Intel(R)Xeon(R)Platinum 8474C. The text-side feature extractor of the present invention uses RoBERTa-wwm-Large, and the visual-side feature extractor uses ViT-H / 14. The dimensions of the generated original feature embedding vectors are all 1024, and the dimension of the encrypted feature embedding vector is 1027. The parameters of the basic model are 958M in total, and the number of Pseudo prompts in the General Adapter is 15.

[0112] Figure 6 is a schematic diagram showing a comparison of the recall rates of various models when performing a text-by-image search task on CN-Flickr30K according to an exemplary embodiment. The comparison models are the Chinese text-by-image pre-training large model BriVL, Wukong ViT-L / 14 and CN-CLIP ViT-H / 14 ,Depend on Figure 6The experimental results show that the overall recall rate of the present invention when performing image-based text search tasks is improved by about 1.4% compared with the previous method.

[0113] Figure 7 is a schematic diagram showing a comparison of the recall rates of various models when performing a text-to-image search task on CN-COCO according to an exemplary embodiment. From the experimental results, it can be seen that the overall recall rate is improved by about 1.06%.

[0114] Figure 8 It is a schematic diagram of the accuracy of each model when performing a zero-sample classification task of an image on a Chinese image classification dataset according to an exemplary embodiment, and the dataset is the Chinese version of Cifar-10, Cifar-100 and Caltech-101.

[0115] Depend on Figure 8 The experimental results show that the accuracy of the zero-sample classification task of the present invention on the Chinese image classification dataset is improved by about 1.23% on average compared with the previous method on each dataset. In the experiment of testing the retrieval model, a single text and a picture can be regarded as a file, and the number of the two types of files is equal, and the dimension of the encrypted feature embedding vector is 1027.

[0116] Fig. 9 It is a schematic diagram showing how the time consumption of a feature extraction model in generating encrypted feature embedding vectors of files varies with the number of files according to an exemplary embodiment, and the number of files processed in each batch is 128.

[0117] Fig.10 It is a schematic diagram showing how the time consumption of a feature extraction model when constructing an index using encrypted feature embedding vectors of files varies with the number of files according to an exemplary embodiment, wherein the number of approximate neighbors of the HNSW algorithm is set to 5, and the maximum number of connections between each node and the remaining nodes is 16.

[0118] Fig.11 It is a schematic diagram showing how the time consumption of different models for searching for user query vectors on an established index structure varies with the number of files according to an exemplary embodiment. The comparison models use text semantics-based ciphertext retrieval schemes SSSW2 and SSRB2, and the dimensions of the text feature embedding vectors they generate are both 768.

[0119] according to Fig. 9 , Fig.10 and Fig.11 It can be seen that the time consumption of the retrieval model of the present invention when performing approximate nearest neighbor search of vectors on the established index structure varies very stably with the increase in the number of files. Compared with the other two solutions, the more files there are, the more obvious the speed advantage of the present invention in performing retrieval operations.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0121] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A privacy-preserving cross-modal image-text retrieval method for Chinese users, characterized in that: include: In response to receiving the graphic data sent by the user, extracting features from the graphic data using the feature extraction model deployed thereon, and embedding and mapping the features of the extracted graphic data into the same semantic space; Encrypt the text and pictures in the user's graphic data using text encryption technology and image encryption technology respectively, and encrypt the features of the extracted graphic data using asymmetric scalar product preservation encryption technology, and send the encrypted graphic data and the encrypted graphic data feature set to the cloud server; In response to a query request initiated by a user, feature extraction and encryption are performed on the query content input by the user to obtain an encrypted query trapdoor and send it to the cloud server; In response to receiving the encrypted query result returned by the cloud server, the encrypted query result is decrypted and the decrypted plaintext query result is returned to the user.

2. The method according to claim 1, characterized in that Also includes: The feature extraction model is based on the Chinese multimodal pre-trained large model CN-CLIP with all original weight parameters frozen, and is constructed by adding a General Adapter module in front of the text end of the original basic model. The GeneralAdapter model includes a set of learnable vector pseudo prompts and a model component Prompt Encoder that can map the learnable vector pseudo prompts to a specific semantic space.

3. The method according to claim 2, characterized in that The training process of the feature extraction model includes: For the image-text retrieval task and the zero-shot classification task of an image, the training methods of contrastive learning and multi-label learning are combined respectively, and a large-scale Chinese image-text dataset in a general field is used to train the added General Adapter module.

4. The method according to claim 3, characterized in that: The method of encrypting the features of the extracted graphic data using an asymmetric scalar product preserving encryption technique also includes: Through a random key generation algorithm, keys M1, M2 and S are obtained, wherein M1 and M2 are random reversible matrices of dimension (n+3)×(n+3), S is a random vector of dimension 1×(n+2), and the value of the random vector S is composed of randomly distributed 0s and 1s, wherein all keys are open to access and use rights for the private server; For the image and text data feature E = {e1, e2, ..., e m } in the i-th feature e i , generate a random number ε i , e i becomes Use key S to i Decompose into e′ i and e″ i , the principle of decomposition is: if S[j] = 0, then e′ i [j]=e″ i [j], otherwise, e′ i [j]+e″ i [j] = e i [j]; Use keys M1 and M2 to encrypt e′ i and e″ i Encryption, feature e i is encrypted as {e′ i M1,e″ i M2}, denoted as 5. The method according to claim 4, characterized in that In response to a query request initiated by a user, feature extraction and encryption are performed on the query content input by the user to obtain an encrypted query trapdoor and send it to the cloud server, including: In response to the query request initiated by the user, feature extraction is performed on the query content input by the user, and the feature embedding E Q , embed the features of user queries into E according to random numbers Q becomes The expression is: in, is the transformed feature embedding, r and t are random numbers generated by the private server; Use key S to Decompose into E′ Q and E″ Q , the decomposition principle is: for j∈[1,n+2], if S[j]=1, then E′ Q [j]=E″ Q [j]; otherwise, it is E′ Q [j]+E″ Q [j]=E Q [j]; Use the keys separately and For E′ Q and E″ Q Encryption is performed to obtain the encrypted query trapdoor, the expression is: in, A trapdoor for encrypted queries.

6. A privacy-preserving cross-modal image-text retrieval method for Chinese users, comprising: In response to receiving the encrypted graphic data and the encrypted graphic data feature set sent by the private server, using the retrieval model deployed thereon to construct a hierarchical graph-based index for the encrypted data feature set; In response to receiving the encrypted query trapdoor sent by the private server, performing an approximate nearest neighbor search for the query trapdoor in the established index structure to obtain the most appropriate encrypted query result; The most suitable encrypted query result is returned to the private server.

7. The method according to claim 6, characterized in that The method of using the retrieval model deployed thereon to construct a hierarchical graph-based index for the encrypted data feature set includes: Create an empty graph structure, treat each encrypted data feature embedding as a vector node, and insert nodes one by one. Each new node will be randomly assigned a level. The higher the level, the fewer nodes there are. When inserting a new node, start from the highest level and use greedy search to find the nearest neighbor of the level to which the current node belongs, and repeat this process at lower levels to determine more precise neighbor nodes; After finding the nearest neighbor nodes, the inserted new node is connected to these neighbor nodes and the graph structure is updated.

8. The method according to claim 7, characterized in that In response to receiving the encrypted query trapdoor sent by the private server, performing an approximate nearest neighbor search for the query trapdoor in the established index structure to obtain the most suitable encrypted query result, including: When searching, start from the entry node of the highest level in the hierarchical graph and use greedy search to find the node in the current level that is closest to the query vector; After finding the node closest to the query vector in the current layer, search downward layer by layer, repeating this process at each layer to keep getting closer to the query vector; After reaching the bottom layer, an exact search is performed to find the node closest to the query vector.

9. The method according to claim 8, characterized in that Also includes: During the search process, the cosine distance is used as the distance between the query trapdoor and the vector node on the index structure, and the calculation expression is: Wherein, L is the cosine distance, and the inverse of the cosine distance represents the cosine similarity between the encrypted query trapdoor and the ciphertext data corresponding to the vector node on the index structure; The ciphertext data corresponding to the vector node closest to the encrypted query trapdoor is taken as the most suitable encrypted query result.

Images