A data privacy protection method for cross-modal retrieval

Abstract

The present application relates to a kind of data privacy protection methods of cross-modal retrieval.Data owner has paired image-text data, or only has the data of certain modality of image or text, in the case where data is kept locally, multiple data owners train a cross-modal representation learning network with cloud server.Data owner extracts the representation of respective data by the completed model of training, and the encrypted representation is stored with original data in cloud server end and is carried out by it Security clustering.When inquirer queries text by image or queries image by text, cloud server distributes the completed representation learning model of training, and inquirer sends the encrypted query feature to cloud server after extracting the query feature, and returns by cloud server retrieval similar result and return.The present application constructs the aggregation function encryption algorithm of protection model parameter and user weight, provides data privacy protection to cross-modal representation learning model, and realizes the security clustering and security inner product calculation on cloud server.

Description

Technical Field

[0001] This invention relates to a data privacy protection method for cross-modal retrieval. Background Technology

[0002] With the rapid development of artificial intelligence and big data, multimedia data is exploding on the internet, and its forms are becoming increasingly diverse, including images, text, video, and audio. Information retrieval, as a technology frequently used in daily life, has evolved beyond single-modal data retrieval; people now seek cross-modal retrieval through the interaction of features from different modalities. Although these data present themselves in different forms, they are semantically interconnected. Cloud computing, as another significant breakthrough in information technology, can provide on-demand self-service, ubiquitous network access, and elastic and rapid resource management. In recent years, an increasing number of users have chosen to outsource computing and data processing to cloud platforms.

[0003] Cross-modal data retrieval in a cloud computing environment involves an unavoidable issue—privacy protection. Images and text often carry sensitive information, and different users do not want to disclose their sensitive information when performing cross-modal retrieval tasks. Therefore, how to conduct cross-modal retrieval tasks while protecting privacy is also crucial.

[0004] Currently proposed privacy-preserving cross-modal retrieval technologies can be mainly divided into two categories: data protection through encryption schemes and data protection through generative adversarial methods. Mainstream encryption schemes include homomorphic encryption, secure multi-party computation, and differential privacy, which encrypt the data before it is used in the training of cross-modal representation learning models. Generative adversarial data protection works by generating a training dataset that is difficult for a discriminator to distinguish from noise, replacing the original dataset for training, thus protecting the security of the original data.

[0005] Wang et al. [1] proposed a privacy protection method related to CCA [2], which uses additive homomorphic encryption and obfuscated circuits to support various arithmetic operations on encrypted real numbers, and then uses secure CCA technology to complete cross-modal retrieval tasks. Guo et al. [3] combined collective matrix factorization and homomorphic encryption to construct an efficient and accurate scheme to achieve cross-modal retrieval without losing any sensitive information. Zhang et al. [4] proposed to train cross-modal models by generating some adversarial data in order to protect the data. Wang et al. [5] proposed a dual-branch feature extraction network based on convolutional neural networks, which can extract features when text and encrypted image RGB channel values ​​are shuffled, and then learn modal relationships. Zhu et al. [6] proposed a deep adversarial hashing model to capture modal relationships, added a hierarchical hash index structure, and provided effective data organization based on cross-modal hash code.

[0006] With the rise of federated learning, scholars have begun to combine federated learning with cross-modal retrieval, which solves the problem of data dispersion and also the problem of data privacy protection. Zong et al. [7] first combined federated learning with cross-modal retrieval, but did not consider that uploading the model would also lead to the leakage of data information. The training sets of existing cross-modal retrieval models all use paired multimodal data, while in real life, data of different modalities may be distributed among different users. Moreover, most existing solutions combine differential privacy and homomorphic encryption technology to solve the security problem of federated learning, but the use of homomorphic encryption technology is time-consuming and does not take into account the security of user weights.

[0007] References:

[0008] [1] Liu Libo, Gou Tingting. Cross-modal retrieval integrating deep canonical correlation analysis and adversarial learning [J]. Computer Science, 2021, 48(09):200-207.

[0009] [2]Wang Q, Hu S, Du M, et al. Learning privately: Privacy-preservingcanonical correlation analysis for cross-media retrieval[C] / / IEEE INFOCOM2017-IEEE Conference on Computer Communications. IEEE, 2017: 1-9.

[0010] [3]Guo C, Jia J, Jie Y, et al. Enabling secure cross-modal retrievalover encrypted heterogeneous IoT databases with collective matrixfactorization[J]. IEEE Internet of Things Journal, 2020, 7(4): 3104-3113.

[0011] [4]Zhang PF, Bai G, Yin H, et al. Proactive privacy-preserving learning for cross-modal retrieval[J]. ACM Transactions on InformationSystems, 2023, 41(2): 1-23.

[0012] [5]Wang Z, Qin J, Xiang X, et al. A privacy-preserving cross-mediaretrieval on encrypted data in cloud computing[J]. Journal of InformationSecurity and Applications, 2023, 73: 103440.

[0013] [6]Zhu L, Song J, Yang Z, et al. DAP2CMH: Deep adversarial privacy-preserving cross-modal hashing[J]. Neural Processing Letters, 2022, 54(4):2549-2569.

[0014] [7]Zong L, Xie Q, Zhou J, et al. FedCMR: Federated Cross-ModalRetrieval[C] / / Proceedings of the 44th International ACM SIGIR Conference onResearch and Development in Information Retrieval. 2021: 1672-1676.. Summary of the Invention

[0015] The purpose of this invention is to provide a data privacy protection method for cross-modal retrieval. From the perspective of data privacy protection, a secure cross-modal retrieval framework is constructed. This framework allows multiple data owners with inconsistent data modalities and multiple queryers with retrieval needs to complete cross-modal retrieval with the powerful computing power of cloud servers without disclosing any data. Simultaneously, for the secret original data stored on the cloud server, the framework also provides a secure clustering technique, offering support for secure clustering of encrypted original data and cross-modal representations, simplifying global retrieval to intra-class retrieval, thereby improving retrieval efficiency. Furthermore, this invention proposes a function-hidden inner product function encryption algorithm, which, when applied to the data retrieval process, can guarantee the security of the retrieval data.

[0016] To achieve the above objectives, the technical solution of this invention is: a data privacy protection method for cross-modal retrieval, which constructs a secure cross-modal retrieval framework. This framework allows multiple data owners with inconsistent data modalities and multiple queryers with retrieval needs to complete cross-modal retrieval with the powerful computing power of cloud servers without disclosing any data. Simultaneously, for the secret original data stored on the cloud server, the framework also provides a secure clustering technique, offering support for secure clustering of encrypted original data and cross-modal representations, simplifying global retrieval to intra-class retrieval. Furthermore, a function-hidden inner product function encryption algorithm is proposed and applied to the data retrieval process to ensure the security of the retrieval data.

[0017] In one embodiment of the present invention, the cross-modal retrieval framework includes two parts: a cross-modal representation learning model and a retrieval model, involving four parties: content owner, cloud server, trusted third party, and querier. The specific implementation is as follows;

[0018] (1) The cloud server randomly initializes the cross-modal representation learning model and distributes the cross-modal representation learning model according to the content owner attributes. Each content owner trains the cross-modal representation learning model locally.

[0019] (2) A trusted third party distributes relevant keys to the cloud server and the content owner. The content owner encrypts the parameters of the cross-modal representation learning model according to the key. The cloud server receives the key with respect to the user weight to aggregate the cross-modal representation learning models of each content owner. Then it distributes the aggregated cross-modal representation learning models to each content owner. Steps (1) and (2) are repeated until the network converges.

[0020] (3) The content owner extracts cross-modal representations from local data based on the cross-modal representation learning model trained in step (2), and encrypts the cross-modal representations using a key from a trusted third party; at the same time, the original data is encrypted using a symmetric encryption algorithm; and the encrypted pairs of data and representations are stored on a cloud server.

[0021] (4) The cloud server uses the improved SOM algorithm to cluster the stored encrypted representations. After the clustering is completed, the content owner uses a key from a trusted third party to re-encrypt the previous encrypted representation and update the encrypted representation stored on the cloud server for secure retrieval.

[0022] (5) When a querying user initiates a query request to the cloud server, the cloud server will send the cross-modal representation learning model to the querying user; the querying user encrypts the extracted representation using a key from a trusted third party, generates a Trapdoor, and sends it to the cloud server; the cloud server retrieves similar results and returns them to the querying user, who then decrypts the original data using the content owner's key to complete a retrieval process.

[0023] In one embodiment of the present invention, in step (1), the method for owners of different attributes, i.e., different modal content, to jointly train the cross-modal representation learning model is as follows:

[0024] The cross-modal representation learning model consists of two sub-networks. Image data is represented by the image sub-network, which uses a pre-trained VGG19 network as the base network for extracting image features. Text data is represented by the text sub-network, which uses a pre-trained Word2Vec model and a TextCNN network for text feature extraction. These two sub-networks serve as pre-trained networks to learn the original semantic representation of the image and text. Then, two fully connected layers with shared weights are connected for cross-modal representation learning, and finally, a linear classification layer is connected to predict the classification of its output, thereby learning information from the label space. Different algorithms are used for cross-modal representation learning for different content owners. The algorithm for content owners who own image-text pairs is as follows:

[0025] (1)

[0026] in, This represents minimizing three loss functions. and This represents two hyperparameters, indicating the weights for different losses. This represents the discriminative loss that measures the label space, used to preserve the semantic similarity between the mapped features and the original data, as well as the distinguishability between different categories of data; its algorithm is as follows:

[0027] (2)

[0028] Represents a linear classification layer. and These represent the common subspace features of the image and text obtained through the image subnetwork and text subnetwork, respectively, after passing through a linear classification layer. Predictive classification and data labels The similarity between the two is measured using the Frobenius norm. This represents the feature dimension of the common subspace.

[0029] This represents the modality invariance loss, used to learn the commonalities of multiple modes and eliminate cross-modal differences; its algorithm is as follows:

[0030] (3)

[0031] The semantic discrimination loss is used to measure the discrimination loss of features in the common subspace, including intra-modal loss and inter-modal loss; its algorithm is as follows:

[0032]

[0033] (4)

[0034]

[0035] in, , These represent the common subspace features of different images and texts, respectively. Represents the cosine similarity function. This indicates an indicator function, which is represented as 1 if the two data points within the function come from the same category, and as 0 otherwise.

[0036] For content owners who only own the image, the algorithm is as follows, where This represents the discriminative loss that measures the label space, but only includes the image portion; This represents semantic discrimination loss, but only includes intra-modal image loss;

[0037] (5)

[0038] For content owners who only possess the text, the algorithm is as follows, where This represents the discriminative loss that measures the label space, but only includes the text portion; This represents semantic discrimination loss, but only includes intra-text modality loss;

[0039] (6).

[0040] In one embodiment of the present invention, in step (2), to protect the data security of the cross-modal representation learning model training process, an aggregation function encryption algorithm for protecting weights is designed, so that multiple local cross-modal representation learning model parameters and user weights are aggregated by the cloud server in an encrypted state to obtain the plaintext result, i.e., the updated cross-modal representation learning model parameters; the aggregation function encryption algorithm for protecting weights is implemented as follows:

[0041] Step 1: A trusted third party generates a master private key and master public key pair based on security parameters. Among them, the master public key Contains cyclic groups Information, This represents a cyclic group. The generators of the cyclic group With model Remaining class addition isomorphism, and a randomly generated vector ,in Represents a random integer. This indicates the dimension of the client-side cross-modal representation learning model parameters. This indicates mapping elements to a cyclic group; corresponding to The weight of each client is used to generate a random value. Combined into an n-dimensional vector Master private key , Let n represent an n-dimensional vector whose elements are in the additive class of the remainder. middle;

[0042] Step 2: Before generating the decryption key containing weights, the trusted third party needs to calculate the weight vector from the attribute information uploaded by the client. Among them, the client The weights are represented as , weight Mapping to the grouping of the remaining classes In, combined with the master private key Generate decryption key Finally, a trusted third party will Send to the cloud server;

[0043] Step 3: A trusted third party, based on the attribute information of each client, combines... and ,generate Send to the One client, of which Represents the modular function;

[0044] Step Four: Each client received the key. Then, the parameters of the locally trained cross-modal representation learning model are... Encryption, among which To represent the length of the model parameter vector, the model parameters are first mapped to the residual class group. , making Then obtain the ciphertext from each client. ;Will The encrypted combination of each client, uploaded encrypted text To the cloud server;

[0045] Step 5: The cloud server receives the decryption key. and ciphertext from the client Then, in the ciphertext Add the master key to the last column of the matrix Then calculate the intermediate value. As shown below:

[0046] (7)

[0047] in The aggregated model parameters in the recurrent group dimensional mapping vector, finally to Calculate the discrete logarithm to obtain the aggregate vector , here That is, the first parameter in the aggregated model parameter vector. dimension.

[0048] In one embodiment of the present invention, in step (4), the stored encrypted representations are clustered by the improved SOM algorithm, that is, by combining the inner product function encryption algorithm with function hiding, the encrypted representations are clustered by the SOM model.

[0049] In one embodiment of the present invention, the SOM model implements clustering in the following way:

[0050] Step 1: Initialize the weight vectors of each neuron ,in The number of neurons, For the dimensions of the input data;

[0051] Step 2: In each round of training, calculate the neuron most similar to the input using the inner product; this is the winning neuron. Then calculate the... The winning neuron in the round The radius of the center Find the neurons that need updating;

[0052] Step 3: Calculate neurons Update coefficients ,in Indicates the first Learning rate of the round, This refers to neurons. With winning neurons distance, This indicates that the update coefficients decay over time, where time is represented by the round number t. express initial value, The total number of iterations is set initially, and the neuron... The update coefficient will change with the winning neuron. The distance increases and the distance decreases, and it also decreases with the number of rounds. Finally, based on... Update neurons The weight vector; repeat step two until the number of rounds reaches the set value.

[0053] In one embodiment of the present invention, the encryption algorithm for the hidden inner product function is implemented as follows:

[0054] Step 1: A trusted third party determines the security parameters and the known vector length. Randomly generate master and public keys and the data owner's master private key and the queryer's master private key The master private key will be used to encrypt the respective plaintext messages; among which It represents random integers, an invertible matrix. , This represents a size of A random invertible square matrix, where all elements are residue-based additive groups. The value in, i.e. , It can also be divided into two invertible square matrices. and . This represents a size of A random matrix, where all elements are residue additive groups. The value in, i.e. ;

[0055] Step 2: The content owner randomly generates a vector. ,pass and Calculate the encrypted feature vector ,exist After splicing ,in This is the feature vector used for calculation. Finally, it is multiplied by the value in the private key. Obtain the final ciphertext ;

[0056] Step 3: Initialize the weight vector on the cloud server Here, the weight vector represents the neuron vector of the SOM, where The number of neurons, Calculate the input data dimensions ,exist After splicing Finally, multiply it by Obtain the decryption key ;

[0057] Step 4: After receiving the encrypted text uploaded by the content owner, the cloud server calculates... Two parts were obtained. , ; Displaying the similarity between neurons and features is used to find features within a neighborhood radius. Neurons within; Used to update neurons For the learning rate, through Update the neurons in each round until the set number of rounds is completed;

[0058] Step 5: After the SOM model training is complete, the cloud server sends a signal to the content owner, who will then update the encrypted feature values ​​stored on the cloud server. .

[0059] In one embodiment of the present invention, step (5) is specifically implemented as follows: When a querying user initiates a retrieval request to the cloud server, the cloud server first shares the trained cross-modal representation learning model. After the querying user extracts the cross-modal representation, a trusted third party sends an encryption key to the querying user. The querying user encrypts their features according to the key and sends a trapdoor to the cloud server. The cloud server calculates the similarity of each cluster center of the encrypted file features after clustering according to the trapdoor, and then calculates the similarity in the class where the most similar cluster center is located. The result of the name is returned to the querying user; after obtaining the encrypted text, the querying user requests the decryption key from the corresponding data owner based on the source of the encrypted text, and can then obtain the plaintext search data. The similarity calculation between the query features and the search data is performed according to the following steps:

[0060] Step 1: A trusted third party sends the private key generated during secure clustering. and the master key For the user making the query;

[0061] Step Two: Query User Access and Encrypted query features get Send to the cloud server;

[0062] Step 3: After receiving the trapdoor, the cloud server calculates its similarity to the secret features stored on the server; since the features have undergone clustering, the similarity is calculated... Find the class most similar to trapdoor, calculate the search results within that class, and return them.

[0063] Step 4: Query the encrypted search results returned by the user through the cloud server, request the symmetric decryption key from the corresponding user, and finally obtain the search results.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] (1) To address the issue of user data leakage during model training, this invention introduces function encryption technology to design a secure cross-modal representation learning method based on federated learning. A novel aggregation function encryption method is designed to ensure that the server performs aggregation calculations without knowing the user model parameters and weights, thereby achieving secure cross-modal representation learning.

[0066] (2) Unlike traditional cross-modal representation learning methods, this invention leverages the advantages of federated learning to allow for diverse training data. It not only breaks the traditional approach of limiting model training to pairwise multimodal data, but also supports mixed training of single-modal and pairwise multimodal data, which is more in line with the real-world scenario of user modality imbalance.

[0067] (3) To address data security issues in cross-modal retrieval, this invention designs a function-hidden inner product function encryption method to achieve similarity measurement between encrypted multimodal data. This protects data privacy during the retrieval process.

[0068] (4) In addition, in order to improve the efficiency of retrieval, this invention introduces a self-organizing map network and designs an inner product function encryption method to perform secure unsupervised clustering of high-dimensional feature vectors, transforming global retrieval into retrieval of data with the same cluster center, thereby greatly reducing computational costs and improving retrieval efficiency.

[0069] (5) The feasibility and efficiency of the solution have been proven through numerous experiments. Multiple data owners can jointly build a secure cross-modal retrieval framework based on this solution, while protecting the data. Queries can also use this solution to achieve secure retrieval. Attached Figure Description

[0070] Figure 1 It demonstrates the overall framework for secure cross-modal retrieval based on federated learning;

[0071] Figure 2 It demonstrates a cross-modal representation model;

[0072] Figure 3 It demonstrates the process of training a cross-modal representation model by multiple users with data of different modalities;

[0073] Figure 4 It demonstrates the secure retrieval process after clustering by the cloud server. Detailed Implementation

[0074] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0075] This invention provides a data privacy protection method for cross-modal retrieval. It constructs a secure cross-modal retrieval framework that allows multiple data owners with inconsistent data modalities and multiple queryers with retrieval needs to complete cross-modal retrieval using the powerful computing power of cloud servers without disclosing any data. Simultaneously, for the secret original data stored on the cloud server, the framework also provides a secure clustering technique, supporting secure clustering of encrypted original data and cross-modal representations, simplifying global retrieval to intra-class retrieval. Furthermore, it proposes a function-hidden inner product function encryption algorithm, applying it to the data retrieval process to ensure the security of the retrieval data.

[0076] The following are specific implementation examples of the present invention.

[0077] Example 1:

[0078] See Figure 1 The basic structure design of the cross-modal retrieval model includes two parts: a cross-modal representation learning model and a retrieval model. Based on its execution elements, we propose a secure cross-modal retrieval scheme in a cloud computing environment, which mainly includes four parties: content owner, cloud server, trusted third party, and queryer.

[0079] The specific implementation steps are as follows:

[0080] (1) The cloud server randomly initializes the cross-modal representation learning network, distributes the network according to the content owner attributes, and each content owner trains the network locally.

[0081] (2) A trusted third party distributes relevant keys to the cloud server and the content owner. The content owner encrypts the model parameters according to the key. The cloud server receives the key with user weights to aggregate the models of each content owner and then distributes the aggregated models to each content owner. Steps (1) and (2) are repeated until the network converges.

[0082] (3) The content owner extracts cross-modal representations from the local data based on the cross-modal representation learning model trained in step (2), and encrypts the cross-modal representations using a key from a trusted third party. Simultaneously, the original data is encrypted using a symmetric encryption algorithm. The encrypted pairs of data and representations are then stored on a cloud server.

[0083] (4) The cloud server uses an improved SOM algorithm to cluster the stored encrypted representations. After clustering, the content owner uses a trusted third party's key to re-encrypt the previous encrypted representation and update the encrypted representation stored on the cloud server for secure retrieval.

[0084] (5) Just as Figure 1As shown, when a user initiates a query request to the cloud server, the cloud server sends a cross-modal representation learning model to the user. The user then encrypts the extracted representation using a key from a trusted third party, generates a Trapdoor, and sends it to the cloud server. The cloud server retrieves similar results and returns them to the user, who then decrypts the original data using the content owner's key, completing a retrieval process.

[0085] Example 2:

[0086] Suppose we have three content owners possessing large amounts of text or image data: one owner possesses only text data, one possesses only image data, and one possesses pairs of image and text data. Our cross-modal retrieval data privacy protection framework uses a federated learning mechanism and aggregation function encryption algorithm to prevent data privacy leaks during model training via reverse reasoning. Furthermore, during the retrieval process, it improves retrieval efficiency through a SOM model and prevents query data leakage through an inner product function encryption algorithm.

[0087] In Example 1, we briefly described the basic structure required for a data privacy protection framework in cross-modal retrieval. Next, we will further elaborate on the specific implementation details based on Example 1. The key features are as follows:

[0088] (1) Content owners from different modalities jointly train a cross-modal representation learning network.

[0089] like Figure 2 As shown, the cross-modal representation learning model consists of two sub-networks. Image data is represented by the image sub-network, which uses a pre-trained VGG19 network as the base network for extracting image features. Text data is represented by the text sub-network, which uses a pre-trained Word2Vec model and a TextCNN network to extract text features. These two sub-networks serve as pre-trained networks to learn the original semantic representation of the image and text. Then, two fully connected layers with shared weights are connected for cross-modal representation learning, and finally, a linear classification layer is connected to predict the classification of its output, thereby learning information from the label space. This invention employs different algorithms for cross-modal representation learning for different content owners. For content owners who own image-text pairs, the algorithm is as follows:

[0090] (1)

[0091] in, This represents the discriminative loss that measures the label space, used to preserve the semantic similarity between the mapped features and the original data, as well as the distinguishability between different categories of data. The algorithm is as follows:

[0092] (2)

[0093] This represents the mode-invariant loss, used to learn the commonalities of multiple modes and eliminate cross-modal differences. The algorithm is as follows:

[0094] (3)

[0095] This represents the semantic discrimination loss, used to measure the discrimination loss of common subspace features, including intra-modal loss and inter-modal loss. Its algorithm is as follows:

[0096]

[0097] (4)

[0098]

[0099] For content owners who only possess the image, the algorithm is as follows: This represents the discriminative loss that measures the label space, but only includes the image portion. This represents semantic discrimination loss, but only includes intra-modal loss for images.

[0100] (5)

[0101] For content owners who only possess the text, the algorithm is as follows, where This represents the discriminative loss that measures the label space, but only includes the text portion. This represents semantic discrimination loss, but only includes intra-textual modal loss.

[0102] (6)

[0103] (2) Encryption of the aggregation algorithm for protecting weights

[0104] As Figure 3 As demonstrated, to protect data security during the training process of the representation learning model, we designed a weight-protected aggregation function encryption algorithm. This algorithm allows multiple local model parameters and user weights to be aggregated by the cloud server in an encrypted state to obtain the plaintext result, i.e., the updated model parameters. The weight-protected aggregation function encryption algorithm is a special public-key encryption algorithm, and its implementation details are as follows:

[0105] Step 1: A trusted third party generates a master private key and master public key pair based on security parameters. The master public key Includes cyclic groups The information, and a randomly generated vector , This indicates the dimension of the client-side model parameters. This indicates that elements are mapped to the cyclic group. Corresponding to... The weight of each client is used to generate a random value. Master private key .

[0106] Step 2: Before generating the decryption key containing weights, the trusted third party needs to calculate the weight vector from the attribute information uploaded by the client. Combined with the master private key Generate decryption key Finally, a trusted third party will... Send to the cloud server.

[0107] Step 3: A trusted third party, based on the attribute information of each client, combines... and ,generate Send to the One client.

[0108] Step Four: Each client received the key. Then, the parameters of the locally trained model... Encryption, obtaining ciphertext .Will The encrypted combination of each client, uploaded encrypted text To the cloud server.

[0109] Step 5: The cloud server receives the decryption key. and ciphertext from the client Then, in the ciphertext Add the master key to the last column of the matrix Then calculate the intermediate value. As shown below:

[0110] (7)

[0111] in It is the mapping element of plaintext in the cyclic group, and finally... Calculate the discrete logarithm to obtain the aggregate vector .

[0112] (3) Secure clustering algorithm

[0113] After the cross-modal representation learning network is trained, the content owner extracts the cross-modal representations of the local data based on the network. The cloud server stores pairs of original data and cross-modal representations. To reduce the computational cost of retrieval and improve retrieval efficiency, we choose to perform unsupervised clustering of the encrypted cross-modal representations using a self-organizing map neural network (SOM). To ensure data security, the original data and cross-modal representations cannot be stored directly in plaintext on the cloud server. Therefore, we combine a function-hidden inner product function encryption algorithm, enabling the encrypted representations to be clustered by the SOM model.

[0114] (3-1) SOM model:

[0115] SOM (Similar Sample Clustering) is an unsupervised learning neural network for classifying similar samples in a high-dimensional data space. It's a neural network with only an input layer and a competition layer, typically distributed in a two-dimensional format, with connections between adjacent neurons. During training, for each input data point, the neuron in the competition layer with the smallest distance to the input data is identified as the winning neuron. The winning neuron influences neighboring neurons with lateral connection weights, while the learning rate and neighborhood size decrease with each training epoch. The details of its sample clustering implementation are as follows:

[0116] Step 1: Initialize the weight vector ,in The number of neurons, For the dimensions of the input data.

[0117] Step 2: In each round of training, the neuron most similar to the input is calculated using the inner product; this is the winning neuron. The neighborhood radius is also calculated. Find the neurons that need updating.

[0118] Step 3: Calculate and update weights ,according to Update the neuron weights. Repeat step two until the set number of rounds is reached.

[0119] (3-2) Function Hiding Inner Product Encryption (FH-IPE):

[0120] From the perspective of the SOM model clustering process, two types of secure computation need to be implemented: secure inner product calculation and secure vector subtraction. FH-IPE, as a special public-key encryption algorithm, can satisfy both the requirements of secure clustering and secure retrieval. However, the clustering and retrieval services target different entities: the former is the cloud server, and the latter is the query user. To reduce communication overhead, the content owner uses the same encryption key to encrypt the representation during both clustering and retrieval. However, this exposes the query user's private key. Therefore, considering the security of subsequent retrieval, we will perform secondary encryption on the encrypted cross-modal representation after clustering, ensuring that the cloud server cannot know the query user's encryption key. The specific implementation steps are as follows:

[0121] Step 1: A trusted third party determines the security parameters and the known vector length. Randomly generate master and public keys and the master private key , Among them, the invertible matrix , , .

[0122] Step 2: The content owner randomly generates a vector. ,pass and Calculate the encrypted feature vector ,exist After splicing Finally, multiply it by the value in the private key. Obtain the final ciphertext .

[0123] Step 3: Initialize the weight vector on the cloud server ,calculate ,exist After splicing Finally, multiply it by Obtain the decryption key .

[0124] Step 4: After receiving the encrypted text uploaded by the content owner, the cloud server calculates... Two parts were obtained. , . Displaying the similarity between neurons and features is used to find features within a neighborhood radius. Neurons within. Used to update neurons For the learning rate, through Update the neurons in each round until the set number of rounds is completed.

[0125] Step 5: After the SOM model training is complete, the cloud server sends a signal to the content owner, who will then update the encrypted feature values ​​stored on the cloud server. .

[0126] (4) Secure Search

[0127] See Figure 4 When a user initiates a retrieval request to the cloud server, the cloud server first shares the trained cross-modal representation learning model. After the user extracts the cross-modal representation, a trusted third party sends an encryption key to the user. Users can encrypt their features using a key and send a trapdoor to the cloud server. The cloud server calculates the similarity of each cluster center for the encrypted file features based on the trapdoor, and then calculates the similarity ranking within the cluster containing the most similar centers. The results are returned to the querying user. After obtaining the encrypted text, the querying user requests the decryption key from the corresponding data owner based on the source information in the encrypted text, and can then obtain the plaintext search data.

[0128] The similarity between query features and retrieved data is calculated according to the following steps:

[0129] Step 1: A trusted third party sends the private key generated during secure clustering. and the master key For the user making the query.

[0130] Step Two: Query User Access and Encrypted query features get Send to the cloud server.

[0131] Step 3: After receiving the trapdoor, the cloud server calculates its similarity to the secret features stored on the server. Since the features have undergone clustering, the similarity is calculated... Find the class most similar to trapdoor, calculate the search results within that class, and return them.

[0132] Step 4: Query the encrypted search results returned by the user through the cloud server, request the symmetric decryption key from the corresponding user, and finally obtain the search results.

[0133] The features of this invention are:

[0134] From a data privacy perspective, this paper introduces function encryption technology to construct a secure cross-modal retrieval framework. Leveraging the advantages of federated learning, this technology allows for diverse training data, supporting both unimodal and pairwise multimodal training. Simultaneously, by introducing function encryption, a novel aggregation function encryption method is designed to ensure that the server performs aggregation calculations without knowing the user's model parameters and weights, thereby achieving secure cross-modal representation learning. Furthermore, a function-hidden inner product function encryption method is designed, combined with a self-organizing map network, to achieve similarity measurement between encrypted multimodal data while significantly reducing computational overhead and improving retrieval efficiency.

[0135] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A data privacy protection method for cross-modal retrieval, characterized in that, This paper constructs a secure cross-modal retrieval framework that allows multiple data owners with inconsistent data modalities and multiple queryers with retrieval needs to complete cross-modal retrievals with the powerful computing capabilities of cloud servers without disclosing any data. Simultaneously, for the secret raw data stored on the cloud server, the framework also provides a secure clustering technique, supporting secure clustering of encrypted raw data and cross-modal representations, simplifying global retrieval to intra-class retrieval. Furthermore, a function-hidden inner product function encryption algorithm is proposed and applied to the data retrieval process to ensure the security of the query data. The cross-modal retrieval framework comprises two parts: a cross-modal representation learning model and a retrieval model, involving four parties: content owners, cloud servers, trusted third parties, and queryers. The specific implementation is as follows. (1) The cloud server randomly initializes the cross-modal representation learning model and distributes the cross-modal representation learning model according to the content owner attributes. Each content owner trains the cross-modal representation learning model locally. (2) A trusted third party distributes relevant keys to the cloud server and the content owner. The content owner encrypts the parameters of the cross-modal representation learning model according to the key. The cloud server receives the key with respect to the user weight to aggregate the cross-modal representation learning models of each content owner. Then it distributes the aggregated cross-modal representation learning models to each content owner. Steps (1) and (2) are repeated until the network converges. (3) The content owner extracts cross-modal representations from local data based on the cross-modal representation learning model trained in step (2), and encrypts the cross-modal representations using a key from a trusted third party; at the same time, the original data is encrypted using a symmetric encryption algorithm; and the encrypted pairs of data and representations are stored on a cloud server. (4) The cloud server uses the improved SOM algorithm to cluster the stored encrypted representations. After the clustering is completed, the content owner uses a key from a trusted third party to re-encrypt the previous encrypted representation and update the encrypted representation stored on the cloud server for secure retrieval. (5) When a user initiates a query request to the cloud server, the cloud server will send the cross-modal representation learning model to the user. The query retrieves the representation extracted by the user through encryption using a trusted third party's key, generates a Trapdoor, and sends it to the cloud server. The cloud server retrieves similar results and returns them to the querying user. The querying user decrypts the original data using the content owner's key to complete a search process. The encryption algorithm for the hidden inner product function is implemented as follows: Step 1: A trusted third party randomly generates a master public key, a master private key for the data owner, and a master private key for the queryer based on security parameters and the known vector length. The master private key will be used to encrypt the plaintext information of each entity. Step 2: The content owner randomly generates a vector, and calculates the encrypted feature vector using the master public key and the data owner's master private key; finally, multiply it by the random invertible square matrix in the data owner's master private key to obtain the final ciphertext; Step 3: The cloud server initializes the weight vectors of each neuron in the SOM algorithm and obtains the decryption key based on the queryer's master private key; Step 4: After receiving the encrypted text uploaded by the content owner, the cloud server calculates the random invertible square matrix in the data owner's master private key based on the decryption key obtained in Step 3, and splits it into two parts. The first part of the result shows the similarity between neurons and features, which is used to find neurons within the neighborhood radius; the second part of the result is used to update neurons. Step 5: After the SOM algorithm model training is completed, the cloud server sends a signal to the content owner, who will then update the encrypted feature value stored on the cloud server.

2. The data privacy protection method for cross-modal retrieval according to claim 1, characterized in that, In step (1), the implementation method for owners of different attributes, i.e., different modal content, to jointly train the cross-modal representation learning model is as follows: The cross-modal representation learning model consists of two sub-networks. Image data is represented by the image sub-network, which uses a pre-trained VGG19 network as the base network for extracting image features. Text data is represented by the text sub-network, which uses a pre-trained Word2Vec model and a TextCNN network for text feature extraction. These two sub-networks serve as pre-trained networks to learn the original semantic representation of the image and text. Then, two fully connected layers with shared weights are connected for cross-modal representation learning, and finally, a linear classification layer is connected to predict the classification of its output, thereby learning information from the label space. Different algorithms are used for cross-modal representation learning for different content owners. The algorithm for content owners who own image-text pairs is as follows: （1） in, This represents minimizing three loss functions. and These represent two hyperparameters, indicating the weights for different losses; This represents the discriminative loss that measures the label space, used to preserve the semantic similarity between the mapped features and the original data, as well as the distinguishability between different categories of data; its algorithm is as follows: （2） Represents a linear classification layer. and These represent the common subspace features of the image and text obtained through the image subnetwork and text subnetwork, respectively, after passing through a linear classification layer. Predictive classification and data labels The similarity between the two is measured using the Frobenius norm. Represents the feature dimension of the common subspace; This represents the modality invariance loss, used to learn the commonalities of multiple modes and eliminate cross-modal differences; its algorithm is as follows: （3） The semantic discrimination loss is used to measure the discrimination loss of features in the common subspace, including intra-modal loss and inter-modal loss; its algorithm is as follows: （4） in, , These represent the common subspace features of different images and texts, respectively. Represents the cosine similarity function. This indicates an indicator function, which is represented as 1 if the two data points within the function come from the same category, and as 0 otherwise. For content owners who only own the image, the algorithm is as follows, where This represents the discriminative loss that measures the label space, but only includes the image portion; This represents semantic discrimination loss, but only includes intra-modal image loss; （5） For content owners who only possess the text, the algorithm is as follows, where This represents the discriminative loss that measures the label space, but only includes the text portion; This represents semantic discrimination loss, but only includes intra-text modality loss; （6）。 3. The data privacy protection method for cross-modal retrieval according to claim 1, characterized in that, In step (2), to protect the data security of the cross-modal representation learning model training process, an aggregation function encryption algorithm for protecting weights is designed. This allows multiple local cross-modal representation learning model parameters and user weights to be aggregated by the cloud server in an encrypted state to obtain the plaintext result, i.e., the updated cross-modal representation learning model parameters. The aggregation function encryption algorithm for protecting weights is implemented as follows: Step 1: A trusted third party generates a master private key and master public key pair based on security parameters. Among them, the master public key Contains cyclic groups Information, This represents a cyclic group. The generators of the cyclic group With model Remaining class addition isomorphism, and a randomly generated vector ,in Represents a random integer. This indicates the dimension of the client-side cross-modal representation learning model parameters. This indicates mapping elements to a cyclic group; corresponding to The weight of each client is used to generate a random value. Combined into an n-dimensional vector Master private key , Let n represent an n-dimensional vector whose elements are in the additive class of the remainder. middle; Step 2: Before generating the decryption key containing weights, the trusted third party needs to calculate the weight vector from the attribute information uploaded by the client. Among them, the client The weights are represented as , weight Mapping to the grouping of the remaining classes In, combined with the master private key Generate decryption key Finally, a trusted third party will Send to the cloud server; Step 3: A trusted third party, based on the attribute information of each client, combines... and ,generate Send to the One client, of which Represents the modular function; Step Four: Each client received the key. Then, the parameters of the locally trained cross-modal representation learning model are... Encryption, among which To represent the length of the model parameter vector, the model parameters are first mapped to the residual class group. , making Then obtain the ciphertext from each client. ;Will The encrypted combination of each client, uploaded encrypted text To the cloud server; Step 5: The cloud server receives the decryption key. and ciphertext from the client Then, in the ciphertext Add the master key to the last column of the matrix Then calculate the intermediate value. As shown below: （7） in The aggregated model parameters in the recurrent group dimensional mapping vector, finally to Calculate the discrete logarithm to obtain the aggregate vector , here That is, the first parameter in the aggregated model parameter vector. dimension.

4. The data privacy protection method for cross-modal retrieval according to claim 1, characterized in that, In step (4), the stored encrypted representations are clustered using the improved SOM algorithm, that is, by combining the inner product function encryption algorithm with function hiding, the encrypted representations are clustered by the SOM model.

5. The data privacy protection method for cross-modal retrieval according to claim 4, characterized in that, The SOM model implements clustering in the following way: Step 1: Initialize the weight vectors of each neuron ,in The number of neurons. For the dimensions of the input data; Step 2: In each round of training, calculate the neuron most similar to the input using the inner product; this is the winning neuron. Then calculate the... The winning neuron in the round The radius of the center Find the neurons that need updating; Step 3: Calculate neurons Update coefficients ,in Indicates the first Learning rate of the round, This refers to neurons. With winning neurons distance, This indicates that the update coefficients decay over time, where time is represented by the round number t. express initial value, The total number of iterations is set initially, and the neuron... The update coefficient will change with the winning neuron. The distance increases and the value decreases, and it also decreases with the number of rounds; finally, based on... Update neurons The weight vector; repeat step two until the number of rounds reaches the set value.

6. The data privacy protection method for cross-modal retrieval according to claim 5, characterized in that, The specific implementation of the encryption algorithm for the hidden inner product function is as follows: Step 1: A trusted third party determines the security parameters and the known vector length. Randomly generate master and public keys and the data owner's master private key and the queryer's master private key The master private key will be used to encrypt the respective plaintext messages; among which It represents random integers, an invertible matrix. , This represents a size of A random invertible square matrix, where all elements are residue-based additive groups. The value in, i.e. , Split into two reversible square matrices and ; This represents a size of A random matrix, where all elements are residue additive groups. The value in, i.e. ; Step 2: The content owner randomly generates a vector. ,pass and Calculate the encrypted feature vector ,exist After splicing ,in This is the feature vector used for calculation; finally, it is multiplied by the value in the private key. Obtain the final ciphertext ; Step 3: Initialize the weight vector on the cloud server Here, the weight vector represents the neuron vector of the SOM, where The number of neurons, Calculate the input data dimensions ,exist After splicing Finally, multiply it by Obtain the decryption key ; Step 4: After receiving the encrypted text uploaded by the content owner, the cloud server calculates... Two parts were obtained. , ; Displaying the similarity between neurons and features is used to find features within a neighborhood radius. Neurons within; Used to update neurons For the learning rate, through Update the neurons in each round until the set number of rounds is completed; Step 5: After the SOM model training is complete, the cloud server sends a signal to the content owner, who will then update the encrypted feature values ​​stored on the cloud server. .

7. The data privacy protection method for cross-modal retrieval according to claim 6, characterized in that, Step (5) is implemented as follows: When a querying user initiates a retrieval request to the cloud server, the cloud server first shares the trained cross-modal representation learning model. After the querying user extracts the cross-modal representation, a trusted third party sends an encryption key to the querying user. The querying user encrypts their features according to the key and sends the trapdoor to the cloud server. The cloud server calculates the similarity of each cluster center of the encrypted file features after clustering based on the trapdoor, and then calculates the similarity in the class containing the most similar cluster centers. The result of the name is returned to the querying user; after obtaining the encrypted text, the querying user requests the decryption key from the corresponding data owner based on the source of the encrypted text, and can then obtain the plaintext search data. The similarity calculation between the query features and the search data is performed according to the following steps: Step 1: A trusted third party sends the private key generated during secure clustering. and the master key For the user making the query; Step Two: Query User Access and Encrypted query features get Send to the cloud server; Step 3: After receiving the trapdoor, the cloud server calculates its similarity to the secret features stored on the server; Because the features have undergone clustering, through calculation Find the class most similar to trapdoor, calculate the search results within that class, and return them. Step 4: Query the encrypted search results returned by the user through the cloud server, request the symmetric decryption key from the corresponding user, and finally obtain the search results.

Images