A privacy protection cross-modal image-text retrieval method for Chinese users

Abstract

The application provides a cross-modal image-text retrieval method for privacy protection of Chinese users, which comprises the following steps: a user submits image-text data to a private server, image-text features are mapped to the same semantic space by a feature extraction model designed by an efficient parameter fine-tuning method, and encryption is performed by using an asymmetric scalar product technology; the encrypted image-text data and image-text data features are uploaded to a cloud server together, the cloud server constructs a layered image index of the encrypted data; when the user queries, the private server extracts and encrypts the query features, the cloud server performs an approximate nearest neighbor search through the built index structure, returns the encrypted result to the private server for decryption, and finally sends the plaintext result to the user. The application can protect the privacy and security of user data while providing cross-modal retrieval services for image-text data stored by the user on a third-party cloud server, and has high retrieval accuracy and efficiency and low training cost.

Description

Technical Field

[0001] This application relates to the field of cross-modal retrieval, and in particular to a privacy-preserving cross-modal image and text retrieval method for Chinese users. Background Technology

[0002] With the rapid development of the internet and information technology, personal multimedia data, primarily in the form of images and text, has experienced explosive growth, leading to an ever-expanding demand for data storage space. Consequently, more and more users are inclined to store their private multimedia data on third-party cloud servers to save local storage space. However, this also raises concerns about user data privacy and security. How to effectively protect user data privacy while providing accurate and fast cross-modal data retrieval services has become a pressing research topic in the field of privacy protection.

[0003] Furthermore, existing privacy-preserving cross-modal retrieval schemes are designed and tested based on English datasets, which limits their use by Chinese users. While it's possible to translate Chinese user data into English before storing and retrieving it, this wastes computational resources, and machine translation inevitably introduces semantic distortion and errors. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] This application proposes a privacy-preserving cross-modal image and text retrieval method for Chinese users.

[0006] According to a first aspect of the embodiments of this application, a cross-modal image and text retrieval method for privacy protection for Chinese users is provided, including:

[0007] In response to receiving image and text data sent by a user, the feature extraction model deployed on the image and text data is used to extract features from the image and text data, and the extracted features of the image and text data are embedded and mapped in the same semantic space.

[0008] Text encryption and image encryption technologies are used to encrypt the text and images in the user's image and text data, respectively. Asymmetric scalar product preservation encryption technology is used to encrypt the features of the extracted image and text data. The encrypted image and text data and the encrypted image and text data feature set are sent to the cloud server.

[0009] In response to a user's query request, the system extracts and encrypts the user's input query content to obtain an encrypted query trapdoor and sends it to the cloud server.

[0010] Upon receiving the encrypted query results from the cloud server, the system decrypts the encrypted query results and returns the decrypted plaintext query results to the user.

[0011] Optional, also includes:

[0012] The feature extraction model is based on the Chinese multimodal pre-trained large model CN-CLIP, which freezes all original weight parameters. It is constructed by adding a General Adapter module to the text end of the original basic model. The General Adapter model includes a set of learnable vector pseudo prompts and a model component Prompt Encoder that can map the learnable vector pseudo prompts to a specific semantic space.

[0013] Optionally, the training process of the feature extraction model includes:

[0014] For image-text retrieval tasks and zero-shot image classification tasks, the General Adapter module is trained using a large-scale, general-domain Chinese image-text dataset, combining contrastive learning and multi-label learning training methods respectively.

[0015] Optionally, the step of encrypting the features of the extracted image and text data using asymmetric scalar product preservation encryption technology further includes:

[0016] Through a random key generation algorithm, keys M1, M2, and S are obtained, where M1 and M2 are random invertible matrices of dimension (n+3)×(n+3), and S is a random vector of dimension 1×(n+2), and the value of the random vector S consists of randomly distributed 0s and 1s. All keys are open to access and use by the private server.

[0017] For image and text data features E = {e1, e2, ..., e m The i-th feature e in} i Generate random number ε i , will e i Become

[0018] Use key S to e i Decompose into e′ i and e″ i The principle of decomposition is: if S[j]=0, ​​then e′ i [j]=e″ i [j], otherwise, e′ i [j]+e″ i [j]=e i [j];

[0019] Use keys M1 and M2 respectively to pair e′ i and e″ i Encryption is performed, feature e i Encrypted as {e′ i M1,e″ i M2}, denoted as

[0020] Optionally, the step of responding to a user-initiated query request by extracting features and encrypting the user-input query content to obtain an encrypted query trapdoor and sending it to the cloud server includes:

[0021] In response to a user's query request, features are extracted from the user's input query content, and the resulting feature embedding E is obtained. Q Embed the user query features into E based on random numbers Q Become The expression is:

[0022]

[0023] in, The transformed feature embedding is given, where r and t are random numbers generated by the private server.

[0024] Use key S to Decompose into E′ Q and E″ Q The principle of decomposition is: for j∈[1,n+2], if S[j]=1, then E′ Q [j]=E″ Q [j]; otherwise, it is E′. Q [j]+E″ Q [j] = E Q [j];

[0025] Use keys separately and For E′ Q and E″ Q Encryption is performed to obtain an encrypted query trapdoor, the expression of which is:

[0026]

[0027] in, This is an encrypted query trap.

[0028] According to a second aspect of the embodiments of this application, a cross-modal image and text retrieval method for privacy protection for Chinese users is provided, including:

[0029] In response to receiving encrypted image and text data input by the user and an encrypted image and text data feature set sent by the private server, the system uses the retrieval model deployed on it to construct an index based on a hierarchical graph for the encrypted data feature set.

[0030] In response to receiving an encrypted query trapdoor sent by a private server, an approximate nearest neighbor search is performed on the query trapdoor in the existing index structure to obtain the most suitable encrypted query result.

[0031] Return the most relevant encrypted query result to the private server.

[0032] Optionally, the step of constructing a hierarchical graph-based index for the encrypted data feature set using the retrieval model deployed thereon includes:

[0033] Create an empty graph structure, treat each encrypted data feature embedding as a vector node, and insert nodes sequentially. Each new node will be randomly assigned a level; the higher the level, the fewer nodes there are.

[0034] When inserting a new node, start from the highest level and use a greedy search to find the nearest neighbor of the current node's level, and repeat this process at lower levels to determine more precise neighbor nodes;

[0035] After finding the nearest neighbor nodes, the newly inserted node is connected to these neighbor nodes, and the graph structure is updated.

[0036] Optionally, in response to receiving an encrypted query trapdoor sent by a private server, performing an approximate nearest neighbor search for the query trapdoor in the existing index structure to obtain the most suitable encrypted query result includes:

[0037] When performing a search, start from the entry node of the highest layer in the hierarchical graph and use a greedy search to find the node in the current layer that is closest to the query vector.

[0038] Once the node closest to the query vector is found in the current layer, the search proceeds layer by layer downwards, repeating this process at each layer to continuously approach the query vector.

[0039] Once the bottom layer is reached, the node closest to the query vector is found through precise search.

[0040] Optional, also includes:

[0041] During the search process, the cosine distance is used as the distance between the query trapdoor and the vector node in the index structure, and the calculation expression is:

[0042]

[0043] Where L is the cosine distance, and the reciprocal of the cosine distance represents the cosine similarity between the encrypted query trapdoor and the ciphertext data corresponding to the vector nodes on the index structure.

[0044] The ciphertext data corresponding to the vector node closest to the encrypted query trapdoor is taken as the most suitable encrypted query result.

[0045] The technical solutions provided by the embodiments of this application have at least the following beneficial effects:

[0046] This application enhances the model's feature extraction capability for Chinese image and text data by adding a General Adapter consisting of a set of learnable vectors and a trainable encoder for mapping these vectors before the text end of a pre-trained Chinese model. It combines contrastive learning and multi-label learning methods for training, requiring only a small number of parameters to be trained throughout the process. Furthermore, it protects user privacy during data storage and retrieval operations on cloud servers by using asymmetric scalar product-preserving encryption technology. Finally, it improves the efficiency of the retrieval model in matching queries for trapdoors on cloud servers by employing the HNSW vector approximation nearest neighbor search algorithm. While protecting user data privacy, this application demonstrates strong cross-modal retrieval capabilities and zero-shot image classification for Chinese image and text datasets, along with low training overhead. This application can be used for cross-modal retrieval of Chinese image and text data on outsourced servers.

[0047] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0048] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0049] Figure 1 This is a system framework diagram illustrating a privacy-preserving cross-modal image and text retrieval method for Chinese users, according to an exemplary embodiment.

[0050] Figure 2 This is a flowchart illustrating a privacy-preserving cross-modal image and text retrieval method for Chinese users, according to an exemplary embodiment.

[0051] Figure 3 This is a training process of a feature extraction model in a text and image retrieval task, as illustrated in an exemplary embodiment.

[0052] Figure 4 This is the training process of a feature extraction model in a zero-shot classification task of an image, as illustrated in an exemplary embodiment.

[0053] Figure 5 This is a flowchart illustrating a privacy-preserving cross-modal image and text retrieval method for Chinese users, according to an exemplary embodiment.

[0054] Figure 6 This is a schematic diagram illustrating the recall rate comparison of various models when performing an image-to-text search task on CN-Flickr30K, according to an exemplary embodiment.

[0055] Figure 7 This is a schematic diagram illustrating the recall rate comparison of various models when performing a text-to-image search task on CN-COCO, according to an exemplary embodiment.

[0056] Figure 8 This is a schematic diagram illustrating the accuracy of various models when performing a zero-shot image classification task on a Chinese image classification dataset, according to an exemplary embodiment.

[0057] Figure 9 This is a schematic diagram illustrating how the time consumption of a feature extraction model in generating encrypted feature embedding vectors for files varies with the number of files, according to an exemplary embodiment.

[0058] Figure 10 This is a schematic diagram illustrating how the time consumption of a feature extraction model, based on an exemplary embodiment, varies with the number of files when constructing an index using the encrypted feature embedding vectors of the files.

[0059] Figure 11 This is a schematic diagram illustrating how the time taken to search for a user query vector on an existing index structure varies with the number of files, according to an exemplary embodiment. Detailed Implementation

[0060] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0061] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0062] This application proposes a privacy-preserving cross-modal image and text retrieval method for Chinese users, aiming to solve the problem of Chinese community users struggling to perform efficient and accurate cross-modal retrieval of outsourced image and text data from third-party cloud servers, as well as privacy and security issues during data storage and retrieval.

[0063] Figure 1 This is a system framework diagram illustrating a privacy-preserving cross-modal image and text retrieval method for Chinese users, based on an exemplary embodiment, and describing the interaction process between different entities.

[0064] Figure 2 This is a flowchart illustrating a privacy-preserving cross-modal image and text retrieval method for Chinese users according to an exemplary embodiment, including the following steps:

[0065] Step 101: In response to receiving image and text data sent by the user, the feature extraction model deployed on the image and text data is used to extract features from the image and text data, and the extracted features of the image and text data are embedded and mapped in the same semantic space.

[0066] This step describes the processing procedure on a private server.

[0067] Efficient parameter fine-tuning is a highly efficient technique for fine-tuning parameters of large pre-trained models. Because existing large pre-trained models have a massive number of parameters, traditional full-scale fine-tuning is no longer a suitable option for retraining. Efficient parameter fine-tuning typically freezes most or all of the original model's parameters and adds corresponding structures for fine-tuning based on the specific task. During model training, only the weight parameters of the newly added structures need to be updated, which greatly reduces the computational requirements for fine-tuning training. This application utilizes efficient parameter fine-tuning to improve the feature extraction capability of the original model for general and specialized domain image and text data by introducing a small number of trainable modules.

[0068] In one embodiment of this application, the feature extraction model deployed on the private server is based on the existing multimodal pre-trained model CN-CLIP, with all its original weight parameters frozen, and new trainable modules added at specific locations to extract task-related embedding vectors E = {e1, e2, ..., e} that can accurately represent the global semantic features of the image and text data. m (Assume there are a total of m pieces of text and image data, and the dimension of the data feature embedding is n).

[0069] Among them, the new trainable module added at a specific location is the General Adapter module added before the text end of the original base model. The General Adapter model includes a set of learnable vector pseudo-prompts and a model component Prompt Encoder that can map the learnable vectors Pseudo prompts to a specific semantic space.

[0070] It should be noted that in vector space, Pseudo prompts refer to a set of randomly initialized vectors used to guide and prompt the model to perform specific downstream tasks. The Prompt Encoder is a component composed of MLP or LSTM networks, mainly used to encode Pseudo prompts and transform them into a representation suitable for model input.

[0071] In addition, in one embodiment of this application, the training process of the feature extraction model includes:

[0072] After adding a General Adapter module to the existing pre-trained model, the General Adapter module is trained using a large-scale, general-domain Chinese image and text dataset for image retrieval and zero-shot image classification tasks, respectively, combining contrastive learning and multi-label learning training methods. After training, the model can be used to extract features from Chinese image and text data under different tasks.

[0073] Figure 3 This describes the training process of a feature extraction model in an image and text retrieval task, according to an exemplary embodiment.

[0074] like Figure 3 As shown, training for the image and text retrieval task involves h continuously differentiable Pseudo prompts ([P0], [P1], ..., [P...]. h The template is fed into the Prompt Encoder and mapped to obtain the template embedding prompts([V0], [V1], ..., [V2]). hMeanwhile, the Chinese caption set `captions` for the image is fed into the Token Mapping module. This module performs operations such as word segmentation, adding special lexical units, lexical embedding mapping, paragraph encoding, and positional encoding on the text tags, ultimately obtaining the semantic feature embeddings of the image captions: `word embeddings([W0]、[W1]……[W1])`. s Next, prompts and word embeddings are concatenated, and the concatenated result is fed into the Text Encoder to obtain the global semantic feature embeddings of the entire sentence after normalization. Finally, the inner product of these embeddings and the global semantic feature embeddings of the image generated by the Image Encoder is calculated to obtain the cosine similarity between the image caption and the image.

[0075] In one embodiment of this application, a large-scale Chinese image and text dataset is used, combined with a contrastive learning training method to train the added General Adapter module.

[0076] Figure 4 This describes the training process of a feature extraction model in a zero-shot image classification task, as illustrated in an exemplary embodiment.

[0077] like Figure 4 As shown, the training process for zero-shot image classification is similar to that for image-text retrieval. The original image captions are replaced with the image's text labels, which are then mapped to word embeddings. These word embeddings are combined with prompts generated for this task and fed into the Text Encoder to be mapped to sentence embeddings. During this process, the embedding vector of each image is compared with the embedding vectors of all text labels, and the training method is multi-label learning.

[0078] Step 102: Use text encryption and image encryption technologies to encrypt the text and images in the user's image and text data, and use asymmetric scalar product preservation encryption technology to encrypt the features of the extracted image and text data. Send the encrypted image and text data and the encrypted image and text data feature set to the cloud server.

[0079] In one embodiment of this application, the features of the extracted image and text data are encrypted using asymmetric scalar product preservation encryption technology.

[0080] Asymmetric Scalar Product Preserving Encryption (ASPPE) is an encryption technology specifically designed to ensure data privacy and security. This technology primarily allows scalar product calculations to be performed on encrypted data without exposing the original data. It is mainly used for retrieving outsourced data from untrusted third-party cloud servers. This invention uses this technology to provide privacy and security guarantees for users storing and retrieving outsourced data.

[0081] First, user keys are generated. A random key generation algorithm is used to obtain keys M1, M2, and S. M1 and M2 are random invertible matrices of dimension (n+3)×(n+3), and S is a random vector of dimension 1×(n+2), with values ​​consisting of randomly distributed 0s and 1s. All keys are open to access and use by the private server.

[0082] Then, for the image and text data features E = {e1, e2, ..., e m The i-th feature e in} i Generate random number ε i , will e i Become use key S to e i Decompose into e′ i and e″ i The principle of decomposition is: if S[j]=0, ​​then e′ i [j]=e″ i [j], otherwise, e′ i [j]+e″ i [j]=e i [j]; Subsequently, keys M1 and M2 were used to test e′ respectively. i and e″ i Encryption is performed, feature e i Encrypted as {e′ i M1,e″ i M2}, denoted as

[0083] In addition, the private server uses encryption algorithms for images and text respectively to encrypt the user's image and text data, and then packages the encrypted image and text data together with the encrypted image and text data feature set and sends it to the cloud server.

[0084] Step 103: In response to the user's query request, the query content entered by the user is feature extracted and encrypted to obtain an encrypted query trapdoor and sent to the cloud server.

[0085] In this embodiment of the application, when a user wants to query, the feature embedding E obtained from the user's query is... Q The private server generates random numbers r and t, embedding the user's query features into E. Q Become The expression is:

[0086]

[0087] in, The transformed feature embedding is defined by r and t, which are random numbers generated by the private server.

[0088] Then, using key S Decompose into E′ Q and E″ Q The principle of decomposition is: for j∈[1,n+2], if S[j]=1, then E′ Q [j]=E″ Q [j]; otherwise, it is E′. Q [j]+E″ Q [j] = E Q [j](The proportion of decomposition is random).

[0089] Then, use the keys separately. and For E′ Q and E″ Q Encryption is performed, resulting in the following query trap:

[0090]

[0091] in, This is an encrypted query trap.

[0092] Step 104: Upon receiving the encrypted query result from the cloud server, decrypt the encrypted query result and return the decrypted plaintext query result to the user.

[0093] When the private server receives the encrypted query results from the cloud server, the private server decrypts the encrypted query results using the key mentioned above, and then returns the decrypted plaintext query results to the user.

[0094] Figure 5 This is a flowchart illustrating a privacy-preserving cross-modal image and text retrieval method for Chinese users according to an exemplary embodiment, including the following steps:

[0095] Step 201: In response to receiving encrypted image and text data and encrypted image and text data feature set sent by the private server, construct an index based on a hierarchical graph using the retrieval model deployed on it.

[0096] This step describes the processing on the cloud server.

[0097] In this embodiment of the application, the cloud server treats each encrypted data feature as a vector node, and then uses the HNSW algorithm to index and construct the encrypted data features.

[0098] The HNSW (Hierarchical Navigable Small World) algorithm is an efficient vector approximate nearest neighbor search algorithm, primarily used for finding nearest neighbors in high-dimensional spaces, particularly excelling on large-scale datasets. Based on the theory of navigable small world algorithms, this algorithm quickly finds the nearest neighbor node to the query node by constructing and searching a hierarchical graph structure. This application uses this algorithm to achieve efficient search in massive high-dimensional vectors.

[0099] Specifically, the process of indexing and constructing encrypted data features using the HNSW algorithm includes:

[0100] First, an empty graph structure is created, and each encrypted data feature is embedded as a vector node. Nodes are then inserted sequentially, with each new node randomly assigned a level; higher levels have fewer nodes. When inserting a new node, starting from the highest level, a greedy search is used to find the nearest neighbor of the current node's level. This process is repeated at lower levels to determine more precise neighbor nodes. After finding the nearest neighbors, connections are established between the new node and these neighbors, and the graph structure is updated. Finally, the navigation of the hierarchical graph is further optimized by adding connections and adjusting edges.

[0101] Step 202: In response to receiving the encrypted query trapdoor sent by the private server, perform an approximate nearest neighbor search for the query trapdoor in the existing index structure to obtain the most suitable encrypted query result.

[0102] In this embodiment of the application, when a user wants to query, the private server will send an encrypted query trap to the cloud server according to the content of the user's query. The cloud server will perform an approximate nearest neighbor search for the query trap in the pre-built index structure to obtain the most suitable encrypted query result.

[0103] During the search, starting from the entry node at the highest level of the hierarchical graph, a greedy search is used to find the node in the current level that is closest to the query vector. The search is then performed layer by layer downwards, repeating this process at each level to continuously approach the query vector. Upon reaching the lowest level, an exact search is used to find the node closest to the query vector.

[0104] Furthermore, during the search process, the distance between the query trapdoor and the vector nodes in the index structure is calculated using cosine distance, the reciprocal of which represents the cosine similarity between the query and the data file corresponding to the node. The calculation process is as follows:

[0105]

[0106] Where L is the cosine distance.

[0107] In addition, it is necessary to select appropriate random numbers r and ε. i The goal is to minimize the impact on search accuracy while protecting user privacy and security.

[0108] It should be noted that the last dimension of the encrypted data feature embedding vector records the file number of the corresponding ciphertext data. Based on this, the cloud server returns the ciphertext data corresponding to the vector node closest to the query trapdoor to the private server as the most suitable encrypted query result.

[0109] Step 203: Return the most matching encrypted query result to the private server.

[0110] When the private server receives the most suitable encrypted query result from the cloud server, the private server decrypts the encrypted data and returns the plaintext result to the user.

[0111] In one embodiment of this application, to verify the effectiveness of this application, the recall rate of the feature extraction model of this scheme compared with other similar models in the cross-modal image and text retrieval process, as well as the time variation of the retrieval model of this scheme during index construction and search. The hardware environment of the experiment was: GPU: two NVIDIA GeForce RTX 4090D; CPU: 30 vCPU Intel(R) Xeon(R) Platinum 8474C. The text-side feature extractor of this invention is RoBERTa-wwm-Large, and the visual-side feature extractor is ViT-H / 14. The dimension of the generated original feature embedding vectors is 1024, and the dimension of the encrypted feature embedding vectors is 1027. The total number of parameters of the basic model is 958M, and the number of Pseudo prompts in the General Adapter is 15.

[0112] Figure 6 This is a schematic diagram illustrating the recall rate comparison of various models when performing an image-to-text search task on CN-Flickr30K according to an exemplary embodiment. The comparison models are the Chinese image-to-text pre-trained large model BriVL and Wukong. ViT-L / 14 and CN-CLIP ViT-H / 14 ,Depend on Figure 6The experimental results show that the overall recall rate of the present invention is improved by about 1.4% compared with the previous method when performing image-based text search tasks.

[0113] Figure 7 This is a schematic diagram illustrating the recall rate comparison of various models when performing text-to-image search tasks on CN-COCO according to an exemplary embodiment. The experimental results show that the overall recall rate was improved by approximately 1.06%.

[0114] Figure 8 This is a schematic diagram illustrating the accuracy of each model when performing a zero-shot image classification task on a Chinese image classification dataset, according to an exemplary embodiment. The dataset consists of Chinese versions of Cifar-10, Cifar-100, and Caltech-101.

[0115] Depend on Figure 8 Experimental results show that the accuracy of this invention in performing zero-shot image classification tasks on Chinese image classification datasets is improved by an average of approximately 1.23% compared to previous methods across various datasets. In the experiments testing the retrieval model, individual text and images can be considered as a single file, and the number of both types of files is equal, with the encrypted feature embedding vectors having a dimension of 1027.

[0116] Figure 9 This is a schematic diagram illustrating how the time taken by the feature extraction model to generate encrypted feature embedding vectors for files varies with the number of files, according to an exemplary embodiment, where the number of files processed in each batch is 128.

[0117] Figure 10 This is a schematic diagram illustrating how the time consumption of a feature extraction model in constructing an index using the encrypted feature embedding vector of a file varies with the number of files, according to an exemplary embodiment. In this diagram, the number of approximate neighbors of the HNSW algorithm is set to 5, and the maximum number of connections between each node and the other nodes is 16.

[0118] Figure 11 This is a schematic diagram illustrating how the time taken by different models to search for user query vectors on an existing index structure varies with the number of files, according to an exemplary embodiment. The comparison models are SSSW2 and SSRB2, which are text semantic-based encrypted retrieval schemes, and the dimensions of the text feature embedding vectors they generate are both 768.

[0119] according to Figure 9 , Figure 10 and Figure 11 It can be seen that the retrieval model of this invention maintains a very stable time consumption for approximate nearest neighbor search of vectors on an existing index structure as the number of files increases. Compared to the other two schemes, the more files there are, the more obvious the speed advantage of this invention in performing retrieval operations becomes.

[0120] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A privacy-protected cross-modal image-text retrieval method for Chinese users, characterized in that, The method comprises the steps of: In response to receiving the user-sent image-text data, using a feature extraction model deployed on a private server to extract features of the image-text data, and embedding the extracted features of the image-text data in the same semantic space; The training process of the feature extraction model comprises: for the image-text retrieval task and the zero-shot classification task of the picture, respectively combining the training mode of contrast learning and multi-label learning, and using a large-scale general domain Chinese image-text data set to train the added General Adapter module; The text and picture in the image-text data of the user are respectively encrypted using text encryption technology and picture encryption technology, and the extracted features of the image-text data are encrypted using asymmetric scalar product preserving encryption technology, and the encrypted image-text data and the encrypted image-text data feature set are sent to the cloud server; In response to the user initiating a query request, the query content input by the user is extracted and encrypted to obtain an encrypted query trapdoor, and the query trapdoor is sent to the cloud server; In response to receiving the encrypted query result returned by the cloud server, the encrypted query result is decrypted, and the decrypted plaintext query result is returned to the user; The feature extraction model takes a Chinese multi-modal pre-training large model CN-CLIP with frozen original all weight parameters as a base model, and is constructed by adding a General Adapter module in front of the text end of the original base model, wherein the General Adapter module comprises a group of learnable vector pseudo prompts Pseudo prompts and a model component Prompt Encoder capable of mapping the learnable vector Pseudo prompts to a specific semantic space.

2. The method of claim 1, wherein, The feature extraction model takes a Chinese multi-modal pre-training large model CN-CLIP with frozen original all weight parameters as a base model, and is constructed by adding a General Adapter module in front of the text end of the original base model, wherein the General Adapter module comprises a group of learnable vector pseudo prompts Pseudo prompts and a model component Prompt Encoder capable of mapping the learnable vector Pseudo prompts to a specific semantic space. By a random key generation algorithm, a key is obtained , and wherein, and are random invertible matrices of dimension , is a random vector of dimension , and the values of the random vector consist of 0 and 1 from a random distribution, wherein all the keys are open to access and use by the private server; For the text data feature in the first feature , generate a random number , change to ; Using keys Will be decomposed into , if , then ; Respectively use the key and Encrypt and , the feature is encrypted as , denoted by .

3. The method of claim 2, wherein, The feature extraction model takes a Chinese multi-modal pre-training large model CN-CLIP with frozen original all weight parameters as a base model, and is constructed by adding a General Adapter module in front of the text end of the original base model, wherein the General Adapter module comprises a group of learnable vector pseudo prompts Pseudo prompts and a model component Prompt Encoder capable of mapping the learnable vector Pseudo prompts to a specific semantic space. In response to the user initiating the query request, feature extraction is performed on the query content input by the user, and a feature embedding of the user query is obtained The feature embedding of the user query is transformed into According to the random number The expression is: wherein, is the transformed feature embedding, with is a random number generated by the private server; Use key Will be decomposed into , if , then ; otherwise, it is ; Respectively use the key And To encrypt And Get encrypted query trapdoor, the expression is: wherein, is an encrypted query trapdoor.

4. A privacy-preserving cross-modal document image retrieval method for Chinese users based on the method of claim 3, characterized in that, The method comprises the steps of: In response to receiving the encrypted image-text data and the encrypted image-text data feature set sent by the private server, using a retrieval model deployed on the private server to construct an index based on a hierarchical graph for the encrypted data feature set; In response to receiving the encrypted query trapdoor sent by the private server, performing approximate nearest neighbor search for the query trapdoor in the built index structure to obtain the most suitable encrypted query result; Return the most suitable encrypted query result to the private server.

5. The method of claim 4, wherein, The method comprises the steps of: An empty graph structure is created, each encrypted data feature is embedded as a vector node, and each new node is inserted in turn, and each new node is randomly assigned a level, and the higher the level, the fewer the nodes; When inserting a new node, start from the highest level, find the nearest neighbor of the current node in the level through greedy search, and repeat the process in the lower level to determine more accurate neighbor nodes; After finding the nearest neighbor nodes, the new inserted node is connected with these neighbor nodes and the graph structure is updated.

6. The method of claim 5, wherein, The approximate nearest neighbor search is performed for the query trapdoor in the built index structure to obtain the most consistent encrypted query result, including: When searching, starting from the entry node of the highest layer of the hierarchical graph, the greedy search is used to find the node with the closest distance to the query vector in the current layer; When the node with the closest distance to the query vector in the current layer is found, the search is performed layer by layer downwards, and this process is repeated in each layer to continuously approach the query vector; After reaching the bottom layer, the node closest to the query vector is found through the accurate search.

7. The method of claim 6, wherein, Further comprising: In the search process, the cosine distance is taken as the distance between the query trapdoor and the vector node on the index structure, and the expression is calculated as: wherein, is the cosine distance, and the reciprocal of the cosine distance represents the cosine similarity between the encrypted query trapdoor and the ciphertext data corresponding to the vector node on the index structure. The ciphertext data corresponding to the vector node with the closest distance to the encrypted query trapdoor is taken as the most consistent encrypted query result.

Images