A federated cross-modal hash retrieval method, system, medium and device based on prototype privacy encryption

Abstract

The application discloses a federated cross-modal hash retrieval method and system based on prototype privacy encryption, a medium and equipment, and belongs to the technical field of federated learning and cross-modal retrieval. The technical problem to be solved by the application is how to effectively guarantee the privacy and security of data in the process of federated cross-modal hash retrieval and avoid privacy leakage. The technical solution is as follows: constructing a data set; extracting deep features; constructing a federated cross-modal hash retrieval model; constructing a federated cross-modal hash retrieval model based on prototype privacy encryption by using a neural network and a deep learning method; training the model; and training the federated cross-modal hash retrieval model based on prototype privacy encryption in the final data set. The construction of the federated cross-modal hash retrieval model is as follows: prototype privacy security symmetric encryption; personalized knowledge preservation; hash retrieval query; and retrieval query is performed by using the hash code generated by the client, and the average precision mean is used as an evaluation index of retrieval performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of federated learning and cross-modal retrieval, in particular to a federated cross-modal hash retrieval method, system, medium and equipment based on prototype privacy encryption. BACKGROUND

[0002] With the continuous enrichment of multi-modal data types such as images, texts and audios, information retrieval technology is no longer limited to single modal. Cross-modal retrieval aims to retrieve relevant data of another modal using one modal data, breaking the limitation of traditional retrieval technology that can only retrieve within the same modal, and better meeting people's increasingly diversified retrieval needs. In addition, in the face of massive high-dimensional multi-modal data, how to improve retrieval speed and how to reduce data storage consumption are two important problems faced by current cross-modal retrieval. Through hash technology, high-dimensional multi-modal data is encoded into hash codes, which significantly reduces the storage space consumption of retrieval data and speeds up the retrieval process while preserving the semantic similarity between different modal data. Therefore, cross-modal hash has become a key technology to support large-scale cross-modal retrieval.

[0003] The privacy and multi-source heterogeneity of large-scale multi-modal data pose great challenges to traditional centralized cross-modal retrieval. Improving the privacy protection capability of the model in the cross-modal retrieval task has become the focus of researchers. Many multi-modal data in practical applications are highly sensitive and multi-source heterogeneous, which makes it difficult to collect multimedia data centrally in practical applications. For example, a large number of highly sensitive medical images and diagnosis reports in hospitals are usually collected and maintained by different clinical departments, and can only be used by researchers with the consent of patients.

[0004] In recent years, federated learning architecture models have been applied to cross-modal hash retrieval systems. Among them, the method based on the prototype architecture generates global prototypes and local prototypes through the server and client of the federated learning process respectively, and ensures the privacy and security of data in the federated learning process through the interaction of global prototypes and local prototypes. However, this method cannot effectively guarantee the security of data, and attackers can still analyze these prototypes and may infer sensitive information, resulting in privacy leakage.

[0005] Therefore, how to effectively guarantee the privacy and security of data in the process of federated cross-modal hash retrieval and avoid privacy leakage is a technical problem to be solved. SUMMARY

[0006] The technical task of the present application is to provide a federated cross-modal hash retrieval method, system, medium and equipment based on prototype privacy encryption, to solve the problem of how to effectively guarantee the privacy and security of data in the process of federated cross-modal hash retrieval and avoid privacy leakage.

[0007] The technical task of the present application is achieved in the following manner, a federated cross-modal hash retrieval method based on prototype privacy encryption, which is specifically as follows:

[0008] Building a data set: obtaining and organizing public data sets of image modalities and text modalities, dividing each public data set into a training set, a test set and a retrieval set, and combining to generate a final data set;

[0009] Deep feature extraction: the original data of the public data set of the image modality and the text modality are respectively extracted by the pre-training model, and the image modality deep feature and the text modality deep feature are obtained, which are used to reflect the semantics of the modality data;

[0010] Building a federated cross-modal hash retrieval model: using neural network and deep learning method to build a federated cross-modal hash retrieval model based on prototype privacy encryption; Specifically as follows:

[0011] Prototype privacy secure symmetric encryption: taking the image modality deep feature and the text modality deep feature as input, generating prototype information in the process of federated learning, ensuring that the privacy information will not be leaked in the process of information exchange and model parameter updating, and outputting the decrypted global prototype;

[0012] Personalized knowledge preservation: taking the client modality initial parameter as input, obtaining the updated parameter through the update hypernetwork part and the judgment hypernetwork extension part, calculating the difference value according to the updated parameter and the input parameter, and finally outputting the updated parameter, realizing the effective preservation of the previously obtained knowledge and the ability to adapt to new data, and relieving the catastrophic forgetting of hypernetwork personalized parameters;

[0013] Hash retrieval query: the hash code generated by the client is used for retrieval query, and the average precision mean value is taken as the evaluation index of retrieval performance;

[0014] Training model: training the federated cross-modal hash retrieval model based on prototype privacy encryption in the final data set.

[0015] As preferred, the data set is built as follows:

[0016] Obtaining public data set: obtaining and downloading public data set containing image modality and text modality from public network resources as original data set; wherein, the public data set includes FashionVC data set and Ssense data set;

[0017] Data division is performed on the downloaded original data set, specifically as follows:

[0018] Randomly select any part of the image-text pairs in the FashionVC dataset and the Ssense dataset as the test set for performance evaluation;

[0019] Exclude the test set image-text pair data in the FashionVC dataset and the Ssense dataset, and randomly select any part of the remaining image-text pairs as the retrieval dataset for retrieval operation;

[0020] In the retrieval dataset, the remaining image-text pairs in the dataset after removing the test set and the retrieval set are used as the training set for the objective function training to achieve better model retrieval performance.

[0021] As a preferred, the deep feature extraction is as follows:

[0022] Extracting image modal deep features: using a pre-trained model ViT as an image encoder to achieve deep feature extraction of the image modal; the specific steps are as follows:

[0023] Preprocessing the original data of the image modal: uniformly crop and reshape the input image to HxW pixel size, and convert the reshaped image into a flattened two-dimensional plane image block;

[0024] Pre-training ViT image encoder: extracting features of the image modal by stacking multiple Transformer layers, and performing global average pooling operation on all position outputs of the last Transformer layer to obtain the global average pooling operation representation of the entire image;

[0025] Extracting image features: performing linear projection on the average pooling representation output by the image modal to obtain the embedding feature representation of the image, which is a vector used to capture high-level semantic information of the image;

[0026] Extracting deep features of the text modal: using a pre-trained model BERT as a text encoder to achieve deep feature extraction of the text modal; the specific steps are as follows:

[0027] Preprocessing the original data of the text modal: dividing the input text into multiple segments, one word per segment, returning a list, and adding an identifier to the list to convert it into a vector representation; and considering the positional relationship of words in the text sentence;

[0028] Pre-training BERT text encoder: using a multi-head self-attention mechanism network to focus on different parts in different positions of the input sequence to capture complex context relationships, and applying a feedforward neural network after each attention layer to perform nonlinear transformation on the features;

[0029] Extracting text features: applying layer normalization after multi-head self-attention operation and feed-forward neural network operation to obtain text deep feature representation.

[0030] As preferred, the prototype privacy security symmetric encryption is as follows:

[0031] Client local prototype generation: generate local prototypes of image modalities and text modalities in the client respectively, which are used to capture information specific to the client, and the formula is as follows:

[0032]

[0033]

[0034] where x i and y i represent the image modality deep feature and the text modality deep feature on the i-th client respectively; m i represents the number of samples on the i-th client; and represent the image modality hash function and the text modality hash function respectively; and represent the image modality local prototype and the text modality local prototype in the i-th client respectively; and are training parameters;

[0035] Client local prototype encryption: construct the encryption operation of the local prototype for identity authentication to ensure the security of the client generating the local prototype, and the formula is as follows:

[0036]

[0037] where e(·) and ε(·) represent the encryption operation; represents the prototype information of the i-th client; represents the noise encryption corresponding to the i-th client;

[0038] Server local prototype decryption: construct the decryption operation of the local prototype on the server side for identity authentication, and subsequently perform secure aggregation of local prototypes in the server, and the formula is as follows:

[0039]

[0040]

[0041] where d(·) and D(·) represent the decryption operation; represents the client local prototype encryption; auth represents the confirmation information of the server comparing the local prototype; χ represents the key;

[0042] Server-side global prototype encryption: the encryption operation of constructing global prototype for identity authentication is performed to ensure that the server-side aggregated global prototype can be securely transmitted to the client, and the formula is as follows:

[0043]

[0044]

[0045] wherein N represents the number of clients; P gx and P gy represent the global prototypes of image modalities and text modalities; and are the client local prototypes, and the final global prototype of the client is represented as P g ={P gx ,P gy};

[0046] Server-side global prototype encryption: the encryption operation of constructing global prototype for identity authentication is performed to ensure that the server-side aggregated global prototype can be securely transmitted to the client, and the formula is as follows:

[0047] e(P g )=ε(P g +n g );

[0048] auth′=J(χ,P g );

[0049] wherein e(·) and ε(·) represent the encryption operation; P g represents the final global prototype of the client; n g represents the server-side noise encryption; and auth′ represents the authentication signal corresponding to the global prototype;

[0050] Client global prototype decryption: after the encrypted global prototype is received by the client, the client global prototype decryption process is performed to securely decrypt, so that the client receives a secure and complete global prototype; the formula is as follows:

[0051] d(P g )=D(e(P g )-n g )=P g ;

[0052] wherein d(·) and D(·) represent the decryption operation; and e(P g ) represents the server-side encrypted global prototype.

[0053] As preferred, the personalized knowledge is saved as follows:

[0054] Update super network: after the element-wise multiplication and The difference calculation process updates the modal hypernetwork; wherein, represents the image modal hypernetwork; represents the text modal hypernetwork; and represents the hypernetwork embedding vector; and represents the hypernetwork parameter; and respectively represents the image and text modal parameters; and respectively represents the updated image and text modal parameters;

[0055] Judging the hypernetwork expansion: using the hypernetwork expansion strategy to selectively retain and update the previously obtained category knowledge, detecting whether the previously saved category knowledge and the newly uploaded category knowledge are consistent:

[0056] If the category knowledge is consistent, the parameter range will not be expanded to merge the new knowledge.

[0057] As a preferred, the hash retrieval query is to calculate the Hamming distance between the hash codes for similarity retrieval, and the average precision is calculated multiple times for iteration to evaluate the retrieval performance, as follows:

[0058] For each modal, the average precision (AP) under different recall rates is calculated, and the formula is as follows:

[0059]

[0060] Wherein, N represents the total number associated with the retrieval sample; P(m) represents the precision of the first M instances related, when the mth sample is related to the retrieval sample, δ(m)=1; otherwise, δ(m)=0;

[0061] The mean average precision (MAP) is calculated according to the average precision, and the formula is as follows:

[0062]

[0063] Wherein, Q represents the number of retrieval sets; AP(i) represents the ith average precision value;

[0064] For each modal, the average precision (AP) under different recall rates is calculated, and the average precision of each modal is averaged to obtain the mean average precision of each modal.

[0065] More preferably, the training model is as follows:

[0066] Constructing loss function: the goal of the federated cross-modal hash retrieval model based on prototype privacy encryption is to compare and match hash codes with other clients under the premise of encryption and privacy protection, to achieve knowledge sharing and collaborative learning, and to obtain high-quality hash codes, which are jointly optimized through a loss function, and are labeled as L (i) , hash code B (i) , category similarity transformation matrix C (i) , and global prototype P g Variable optimization update; the specific process is as follows:

[0067] Optimization update B (i) , the formula is as follows:

[0068]

[0069] Wherein, represents the image hash function; represents the text hash function, which is used to eliminate semantic gaps and achieve better mapping from multi-modal data to hash codes, and to minimize the difference between hash codes and image hash functions and text hash functions;

[0070] Shared optimization update B (i) , the formula is as follows:

[0071]

[0072] The target function is converted into the operation form of , which helps to solve the statistical heterogeneity between clients; P g represents the final global prototype of the client, which helps to capture important information that may not exist alone in the local category relationship, in addition, by considering the common features shared between different clients, the relationship between data and hash codes is more comprehensively represented;

[0073] Optimization update C (i) , the formula is as follows:

[0074]

[0075] The target function is converted into the operation form of , which helps to maintain the similarity between samples in the original data space and the corresponding hash codes, align the hash codes with the category similarity transformation matrix, and ensure that samples with labels or categories have similar hash codes, maintain the semantic structure and organization of the data;

[0076] Optimization model; use Adam algorithm as the optimization function of the model; wherein, the learning rate parameter is set to 0.0001, and other hyperparameters can use the default values in PyTorch.

[0077] A prototype privacy encryption-based federated cross-modal hash retrieval system, which is used to implement the prototype privacy encryption-based federated cross-modal hash retrieval method as described above; the system comprises:

[0078] A data set construction module for obtaining and organizing public data sets of image modalities and text modalities, dividing each public data set into a training set, a test set and a retrieval set, and combining to generate a final data set;

[0079] A deep feature extraction module for extracting features from raw data of public data sets of image modalities and text modalities respectively through a pre-trained model to obtain image modality deep features and text modality deep features, the image modality deep features and the text modality deep features being used to reflect the semantics of modality data;

[0080] A model construction module for constructing a prototype privacy encryption-based federated cross-modal hash retrieval model using a neural network and a deep learning method; the model construction module comprises:

[0081] A prototype privacy secure symmetric encryption sub-module for taking image modality deep features and text modality deep features as inputs to generate prototype information in a federated learning process, ensuring that private information is not leaked in the process of information exchange and model parameter updating, and outputting a decrypted global prototype; specifically, the image modality deep features and the text modality deep features are taken as inputs to generate a local prototype in a client local prototype generation part, then the local prototype is encrypted in a client local prototype encryption part, the encrypted local prototype is uploaded to a server local prototype decryption part to decrypt the local prototype, then a server global prototype generation part aggregates the decrypted local prototype to generate a global prototype, then a server global prototype encryption part encrypts the global prototype, and finally a client global prototype decryption part decrypts the encrypted global prototype to output the decrypted global prototype;

[0082] A personalized knowledge preservation sub-module for taking client modality initial parameters as inputs, obtaining updated parameters through an update hypernetwork part and a judgment hypernetwork expansion part, calculating a difference value based on the updated parameters and the input parameters, and finally outputting the updated parameters to effectively preserve previously obtained knowledge and adapt to new data, and to alleviate the catastrophic forgetting of hypernetwork personalized parameters; specifically, the client initial parameters are taken as inputs, the difference value between the updated parameters and the input parameters is calculated through the update hypernetwork part, then the judgment hypernetwork expansion part is used to effectively preserve previously obtained knowledge and adapt to new data, the personalized adjustment is made according to the specific tasks and data of each client, the performance and generalization of the model are maximized, the catastrophic forgetting of hypernetwork personalized parameters is alleviated, and finally the difference value updated parameters are outputted;

[0083] The hash retrieval query submodule is used for retrieval query through the hash code generated by the client, and the average precision mean is obtained as an evaluation index of retrieval performance; specifically, the hash code generated by the client is used for similarity retrieval through calculation of the Hamming distance, the average precision (AP) is calculated through multiple iterations, then the average precision mean (MAP) is calculated, and finally the average precision mean is used for evaluation of the retrieval performance.

[0084] The training model module is used for constructing a loss function required in a training process of a prototype privacy encryption federated cross-modal hash retrieval model, and completing optimization training of the prototype privacy encryption federated cross-modal hash retrieval model.

[0085] An electronic device, comprising a memory and at least one processor;

[0086] The memory stores a computer program;

[0087] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the prototype privacy encryption federated cross-modal hash retrieval method as described above.

[0088] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program can be executed by a processor to realize the prototype privacy encryption federated cross-modal hash retrieval method as described above.

[0089] The prototype privacy encryption federated cross-modal hash retrieval method, system, medium and device of the application have the following advantages:

[0090] (1) The application solves the problem of guaranteeing the security of the prototype information from being attacked by attackers and causing information leakage in the federated cross-modal hash retrieval method, thereby guaranteeing the generation of high-quality hash codes and effectively improving the accuracy and efficiency of federated cross-modal retrieval, and providing a more superior solution for the practical application of federated cross-modal retrieval and other fields;

[0091] (2) The application uses ViT and BERT as the modal deep feature extraction encoder, which not only enriches the semantic information of multi-modal data, but also more accurately captures the underlying semantic information of images and texts, thereby enriching the feature semantic information of multi-modal data and improving the accuracy and efficiency of retrieval;

[0092] (3) The application realizes distributed training based on multiple clients in the federated cross-modal process, guarantees the security and privacy protection of intermediate parameters, and avoids the risk of data privacy leakage that may be caused;

[0093] (iv) This invention proposes a prototype privacy-secure symmetric encryption scheme to protect the secure transmission of prototype information during the federated learning process, which enhances information security during the transmission of prototype information and prevents potential attackers from cracking prototype information and leaking privacy data.

[0094] (V) This invention designs a personalized knowledge storage module, which dynamically stores the knowledge previously acquired by each client through the corresponding client hypernetwork, realizes personalized federated learning for the client, and proposes to use the hypernetwork expansion method to eliminate the parametric knowledge forgetting problem.

[0095] (vi) This invention utilizes both local and global prototypes, which can capture the unique data distribution characteristics of each client and reflect the inherent distribution pattern of the client's own data, as well as capture the integrated characteristics of shared knowledge among multiple clients, thereby enhancing the generalization of the model.

[0096] (vii) This invention constructs a model for federated cross-modal hash retrieval based on prototype privacy encryption, ensuring that privacy information will not be attacked and leaked during prototype information exchange and model parameter update, thereby protecting the privacy of sensitive data in the federated learning process.

[0097] (VIII) This invention proposes a neural network structure for prototype privacy encryption in federated cross-modal hash retrieval: the neural network structure ensures that the prototype information is not compromised by attacks during the secure update of model parameters; its core idea is to achieve effective preservation of previously acquired knowledge and the ability to adapt to new data by using symmetric encryption and decryption operations of the global prototype and local prototype during the initial parameter update process of the prototype on the server and client sides, while the intermediate parameters are extended through a super network. This allows participants to protect the privacy of their sensitive data in federated learning, actively participate in model training and knowledge sharing, and dynamically update and adjust during federated learning to adapt to the characteristics and changes of new data, generating high-quality hash codes. This promotes secure and efficient federated learning practices, ensures that the retrieval efficiency of cross-modal hash retrieval is not affected, and improves retrieval accuracy. Attached Figure Description

[0098] The invention will be further described below with reference to the accompanying drawings.

[0099] Appendix Figure 1 A flowchart of a federated cross-modal hash retrieval method based on prototype privacy encryption;

[0100] Appendix Figure 2 This is a schematic diagram of a federated cross-modal hash retrieval method based on prototype privacy encryption;

[0101] Appendix Figure 3 A flowchart illustrating the process of prototype privacy-preserving symmetric encryption;

[0102] Figure 1 is a flow chart of a process for personalized knowledge preservation. Figure 4 Figure 2 is a flow chart of a process for hash retrieval query.

[0103] Figure 3 is a flow chart of a process for personalized knowledge preservation. Figure 5 Figure 4 is a flow chart of a process for hash retrieval query. DETAILED DESCRIPTION

[0104] The prototype privacy encryption-based federated cross-modal hash retrieval method, system, medium and device of the present application are described in detail below with reference to the accompanying drawings and specific embodiments.

[0105] Embodiment 1:

[0106] Figure 1 is a flow chart of a process for personalized knowledge preservation. Figure 1 Figure 2 is a flow chart of a process for hash retrieval query. 2 As shown in Figures 1 and 2, the present embodiment provides a prototype privacy encryption-based federated cross-modal hash retrieval method, which is as follows:

[0107] S1, constructing a data set: obtaining and organizing public data sets of image modalities and text modalities, dividing each public data set into a training set, a test set and a retrieval set, and combining to generate a final data set;

[0108] S2, deep feature extraction: extracting features from the original data of the public data sets of the image modalities and the text modalities through a pre-trained model to obtain image modality deep features and text modality deep features, the image modality deep features and the text modality deep features being used to reflect the semantics of the modality data;

[0109] S3, constructing a federated cross-modal hash retrieval model: constructing a prototype privacy encryption-based federated cross-modal hash retrieval model using a neural network and a deep learning method; specifically as follows:

[0110] S301, prototype privacy secure symmetric encryption: taking the image modality deep features and the text modality deep features as inputs to generate prototype information in the federated learning process, ensuring that privacy information is not leaked in the process of information exchange and model parameter updating, and outputting a decrypted global prototype;

[0111] S302, personalized knowledge preservation: taking the client modality initial parameters as inputs, obtaining updated parameters through the update hypernetwork part and the judgment hypernetwork extension part, calculating the difference between the updated parameters and the input parameters, and finally outputting the updated parameters to achieve effective preservation of previously obtained knowledge and the ability to adapt to new data, and to alleviate the catastrophic forgetting of hypernetwork personalized parameters;

[0112] S303, hash retrieval query: performing retrieval query through the hash code generated by the client, and taking the average precision mean as an evaluation index of retrieval performance;

[0113] S4, training a model: training a federated cross-modal hash retrieval model based on a prototype privacy encryption in the final data set.

[0114] The constructing data set in step S1 of the embodiment is specifically as follows:

[0115] S101, obtaining a public data set: obtaining and downloading a public data set containing an image mode and a public data set containing a text mode from a public network resource as an original data set; wherein the public data set includes a FashionVC data set and a Ssense data set;

[0116] S102, data division on the downloaded original data set, specifically as follows:

[0117] S10201, randomly selecting any part of image-text pairs on the FashionVC data set and the Ssense data set as a test set for evaluating test performance;

[0118] S10202, excluding the test set image-text pair data in the FashionVC data set and the Ssense data set, and randomly extracting any part of the remaining image-text pairs as a retrieval data set for retrieval operation;

[0119] S10203, in the retrieval data set, the remaining image-text pairs of the data set excluding the test set and the retrieval set are used as a training set for target function training to achieve better model retrieval performance.

[0120] The deep feature extraction in step S2 of the embodiment is specifically as follows:

[0121] S201, extracting image mode deep features: using a pre-trained model ViT as an image encoder to realize deep feature extraction of the image mode; specifically as follows:

[0122] S20101, preprocessing the image mode original data: uniformly cropping and reshaping the input image to HxW pixel size, and converting the reshaped image into a flattened two-dimensional plane image block;

[0123] S20102, pre-training ViT image encoder: extracting features of the image mode by stacking multiple Transformer layers, and performing global average pooling operation on all position outputs of the last Transformer layer to obtain the global average pooling operation representation of the entire image;

[0124] S20103, extracting image features: performing linear projection on the average pooling representation output by the image mode to obtain the embedding feature representation of the image, which is a vector for capturing high-level semantic information of the image;

[0125] S202. Extracting Deep Features of Text Modalities: Using the pre-trained model BERT as a text encoder, deep features of text modalities are extracted; specifically as follows:

[0126] S20201. Preprocessing raw text modal data: The input text is divided into multiple segments, each containing one character. A list is returned, and an identifier is added before the list to convert it into a vector representation. The positional relationship of words in the text sentence is also considered.

[0127] S20202, Pre-trained BERT text encoder: It uses a multi-head self-attention mechanism network to focus on different parts at different positions in the input sequence, captures complex contextual relationships, and applies a feedforward neural network after each attention layer to perform non-linear transformation on the features;

[0128] S20203. Extracting text features: After multi-head self-attention operation and feedforward neural network operation, layer normalization is applied to obtain the deep feature representation of the text.

[0129] For example, in the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0130] def extract_image_features(image):

[0131] vit=Vit.load_pretrained_vit()

[0132] image=preprocess_image(image)

[0133] embeddings = vit(image)

[0134] embeddings=normalize(embeddings)

[0135] return embeddings

[0136] def extract_text_features(text):

[0137] bert=Bert.load_pretrained_bert()

[0138] tokens = tokenize(text)

[0139] embeddings = bert(tokens)

[0140] embeddings=normalize(embeddings)

[0141] return embeddings

[0142] Wherein, vit is used as a loaded pre-training model of ViT for image modal to extract features; bert is a loaded pre-training BERT model for text modal to extract features; tokenize is a pre-processing text segmentation; normalize is a normalized feature.

[0143] As shown in the accompanying Figure 3 The prototype privacy security symmetric encryption in step S301 of the embodiment is specifically as follows:

[0144] S30101, client local prototype generation: generate local prototypes of image modal and text modal in the client respectively, which are used to capture information specific to the client, and the formula is as follows:

[0145]

[0146]

[0147] Wherein, x i and y i represent the image modal deep feature and the text modal deep feature on the i-th client respectively; m i represents the number of samples on the i-th client; and represent the image modal hash function and the text modal hash function respectively; and represent the image modal local prototype and the text modal local prototype in the i-th client respectively; and are training parameters;

[0148] S30102, client local prototype encryption: construct the encryption operation of the local prototype for identity verification, to ensure the security of the local prototype generated by the client, and the formula is as follows:

[0149]

[0150] Wherein, e(·) and ε(·) represent the encryption operation; represents the prototype information of the i-th client; represents the noise encryption corresponding to the i-th client;

[0151] S30103, server local prototype decryption: construct the decryption operation of the local prototype on the server side for identity verification, to perform subsequent secure aggregation of local prototypes in the server, and the formula is as follows:

[0152]

[0153]

[0154] where d(·) and D(·) represent decryption operations; represents client-side local prototype encryption; auth represents server-side authentication information corresponding to the local prototype; and χ represents a key;

[0155] S30104, server-side global prototype encryption: an encryption operation is constructed for identity authentication of the global prototype, to ensure that the server-side aggregated global prototype can be securely transmitted to the client, and the formula is as follows:

[0156]

[0157]

[0158] where N represents the number of clients; P gx and P gy represent global prototypes of image modalities and text modalities; and are client-side local prototypes, and the final global prototype of the client is represented as P g ={P gx ,P gy};

[0159] S30105, server-side global prototype encryption: an encryption operation is constructed for identity authentication of the global prototype, to ensure that the server-side aggregated global prototype can be securely transmitted to the client, and the formula is as follows:

[0160] e(P g ) = ε(P g +n g );

[0161] auth' = J(χ, P g );

[0162] where e(·) and ε(·) represent encryption operations; P g represents the final global prototype of the client; n g represents server-side noise encryption; and auth' represents an authentication signal corresponding to the global prototype;

[0163] S30106, client-side global prototype decryption: after the client receives the encrypted global prototype, the client verifies the identity authentication information to ensure that the client receives a secure and complete global prototype; if the identity authentication is successful, the global prototype is decrypted, and the formula is as follows:

[0164] d(P g )=D(P g +n g -n g ) = P g ;

[0165] Where d(·) and D(·) represent decryption operations; e(P g ) represents the global prototype of server-side encryption.

[0166] For example, in the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0167]

[0168]

[0169] Among them, proto is the prototype information; noise is noise encryption; sign and new_sign are both noise encryption of prototype information.

[0170] As attached Figure 4 As shown, the personalized knowledge storage in step S302 of this embodiment is as follows:

[0171] S30201, Updated Hypernetwork: After element-wise multiplication sum The modal hypernetwork is updated during the process of calculating the difference; among which, Represents an image modality hypernetwork; Represents a text modal hypernetwork; and Represents the hypernetwork embedding vector; and Indicates hypernetic parameters; and These represent the image and text modal parameters, respectively. and These represent the updated image and text modal parameters, respectively.

[0172] S30202. Determine Hypernet Expansion: Utilize the hypernet expansion strategy to selectively retain and update previously acquired category knowledge, and check whether the previously saved category knowledge is consistent with the newly uploaded category knowledge.

[0173] If the category knowledge is consistent, the parameter range will not be expanded to incorporate new knowledge.

[0174] For example, in the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0175] img_hyp=up(img_hyp,img_client_model_params_list,idx,img_diff,opt);

[0176] txt_hyp=up(txt_hyp,txt_client_model_params_list,idx,txt_diff,opt);

[0177] Where img_hyp and txt_hyp are the image modality supernetwork and text modality supernetwork, respectively; img_client_model_params_list and txt_client_model_params_list store the parameter lists for each client's image and text modality, respectively; idx is the index of the current client; img_diff and txt_diff store the gradient differences between the image and text model updates, respectively; and up is the function for updating the supernetwork parameters.

[0178] As attached Figure 5 As shown, in step 303 of this embodiment, the hash retrieval query performs similarity retrieval by calculating the Hamming distance of the hash code, and iterates multiple times to calculate the average precision, which is used to evaluate the retrieval performance, as detailed below:

[0179] S30301. For each modality, calculate the average precision (AP) at different recall rates, using the following formula:

[0180]

[0181] Where N represents the total number of instances associated with the retrieved sample; P(m) represents the precision of the top M instances, where δ(m) = 1 when the m-th instance is associated with the retrieved sample; otherwise, δ(m) = 0.

[0182] S30302, Mean Precision (MAP) is calculated based on the mean precision, using the following formula:

[0183]

[0184] Where Q represents the number of retrieval sets; AP(i) represents the i-th average precision value;

[0185] For each modality, the mean precision (AP) at different recall rates is calculated, and the mean precision of each modality is averaged to obtain the mean mean precision of each modality.

[0186] For example, the process described above can be represented in PyTorch as follows:

[0187]

[0188]

[0189] Wherein, torch.mean(count / tindex) is the average precision (AP) calculation; map is the calculation of the mean average precision (MAP).

[0190] The training model in step S4 of the embodiment is specifically as follows:

[0191] S401, constructing a loss function: the goal of the federated cross-modal hash retrieval model based on the prototype privacy encryption is to compare and match the hash code with other clients to realize knowledge sharing and collaborative learning under the premise of encryption and privacy protection, so as to obtain high-quality hash code, which is jointly optimized through the loss function, and the label L (i) , hash code B (i) , category similarity transformation matrix C (i) , and global prototype P g Variable optimization update; Specifically as follows:

[0192] S40101, minimize the optimization update B (i) , the formula is as follows:

[0193]

[0194] Wherein, represents the image hash function; represents the text hash function, which is used to eliminate semantic gap and realize better mapping from multi-modal data to hash code, and minimize the difference between hash code and image hash function and text hash function;

[0195] S40102, share optimization update B (i) , the formula is as follows:

[0196]

[0197] The target function is transformed into the operation form of , which is helpful to solve the statistical heterogeneity between clients; P g represents the final global prototype of the client, which is helpful to capture important information that may not exist in the local category relationship alone, in addition, by considering the common features shared between different clients, the relationship between data and hash code is more comprehensively represented;

[0198] S40103, optimization update C (i) , the formula is as follows:

[0199]

[0200] The target function is transformed into The operation form of the loss function helps to keep the similarity between the samples in the original data space and the corresponding hash code, align the hash code with the category similarity transformation matrix, ensure that the samples with labels or categories have similar hash codes, maintain the semantic structure and organization of the data, and the like.

[0201] For example, the loss function described above is expressed in code in PyTorch as follows:

[0202] Loss1=opt.alpha*torch.sum(torch.pow(B-F,2)+torch.pow(B-G,2)

[0203] Loss2=opt.beta*torch.pow(train_L-torch.matmul(B,P.t()),2)

[0204] Loss3=opt.gamma*torch.pow(train_L-torch.matmul(B,C.t()),2)

[0205] Wherein, opt.alpha, opt.beta and opt.gamma are all hyperparameters of the loss function.

[0206] S402, optimizing the model; using the Adam algorithm as the optimization function of the model; wherein the learning rate parameter is set to 0.0001, and other hyperparameters can use the default values in PyTorch.

[0207] The embodiment selects the MAP value as an evaluation index.

[0208] For example, the optimization function described above is expressed in code in PyTorch as follows:

[0209] optimizer=torch.optim.Adam(model.parameters(),lr=learning_rate)

[0210] Wherein, torch.optim.Adam is the Adam optimization function embedded in pytorch; model.parameters() is a parameter set for model training; and learning_rate is the learning rate.

[0211] The model of the embodiment is experimented on two public data sets of FashionVC and Ssense, and the experiment shows that the model of the embodiment is superior to the effect of the existing model, and the comparison of the experimental results is specifically shown in the following table: the model of the present application and the existing model are compared, and it can be seen that the method of the present application is the best in performance compared with other methods. Among them, the FedAvg model comes from the literature "Communication-Efficient Learning Of Deep Networks From Decentralized Data", and the FedCMR model comes from the literature "FedCMR: Federated Cross-modal Retrieval".

[0212]

[0213] As shown in the above table, compared with the existing model, the model in the embodiment is the best in performance.

[0214] Embodiment 2:

[0215] The embodiment provides a federated cross-modal hash retrieval system based on prototype privacy encryption, which is used to implement the federated cross-modal hash retrieval method based on prototype privacy encryption as in embodiment 1; the system comprises:

[0216] A data set construction module is configured to obtain and organize public data sets of image modalities and text modalities, divide each public data set into a training set, a test set and a retrieval set, and combine to generate a final data set;

[0217] A deep feature extraction module is configured to extract features from original data of public data sets of image modalities and text modalities respectively by using a pre-trained model, to obtain image modality deep features and text modality deep features, and the image modality deep features and the text modality deep features are used to reflect the semantics of the modality data;

[0218] A model construction module is configured to construct a federated cross-modal hash retrieval model based on prototype privacy encryption by using a neural network and a deep learning method; the model construction module comprises:

[0219] The prototype privacy security symmetric encryption submodule is configured to take image modal deep features and text modal deep features as inputs, generate a prototype in a federated learning process, ensure that privacy information is not leaked in the process of information exchange and model parameter updating, and output a decrypted global prototype. Specifically, the image modal deep features and the text modal deep features are taken as inputs, a local prototype is generated in a client local prototype generation part, then the local prototype is encrypted in a client local prototype encryption part, the encrypted local prototype is uploaded to a server local prototype decryption part to decrypt the local prototype, then a server global prototype generation part aggregates the decrypted local prototypes to generate a global prototype, then a server global prototype encryption part encrypts the global prototype, and finally a client global prototype decryption part decrypts the encrypted global prototype to output the decrypted global prototype.

[0220] The personalized knowledge preservation submodule is configured to take client modal initial parameters as inputs, obtain updated parameters through an update hypernetwork part and a judgment hypernetwork expansion part, calculate a difference value based on the updated parameters and the input parameters, and finally output the updated parameters, so as to effectively preserve previously obtained knowledge, adapt to new data, and alleviate catastrophic forgetting of hypernetwork personalized parameters. Specifically, the client initial parameters are taken as inputs, the updated parameters are calculated through the update hypernetwork part, and a difference value is calculated based on the updated parameters and the input parameters. Then, the judgment hypernetwork expansion part is used to effectively preserve previously obtained knowledge and adapt to new data. The personalized adjustment is performed based on specific tasks and data of each client, the performance and generalization of the model are maximally improved, catastrophic forgetting of hypernetwork personalized parameters is alleviated, and finally the difference value updated parameters are output.

[0221] The hash retrieval query submodule is configured to perform retrieval query through a hash code generated by a client, and obtain an average precision mean value as an evaluation index of retrieval performance. Specifically, the hash code generated by the client is used for similarity retrieval through calculation of a Hamming distance, an average precision (AP) is calculated through multiple iterations, then an average precision mean (MAP) is calculated, and finally the average precision mean is used to evaluate the retrieval performance.

[0222] The training model module is configured to construct a loss function required in a training process of a prototype privacy encryption federated cross-modal hash retrieval model, and complete optimization training of the prototype privacy encryption federated cross-modal hash retrieval model.

[0223] Embodiment 3

[0224] The embodiment also provides an electronic device, which includes a memory and a processor.

[0225] The memory stores computer execution instructions.

[0226] The processor executes computer-executable instructions stored in the memory such that the processor performs the method of federated cross-modal hash retrieval based on prototype privacy-encryption according to any of the embodiments of the present application.

[0227] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0228] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function, etc.; and the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a secure digital (SD) card, a flash memory card, at least one magnetic disk storage, a flash memory device, or other volatile solid-state memory devices.

[0229] Embodiment 4:

[0230] The embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor to make the processor execute the method of federated cross-modal hash retrieval based on prototype privacy-encryption according to any of the embodiments of the present application. Specifically, a system or device equipped with a storage medium can be provided, and software program codes for realizing the functions of any of the above embodiments are stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0231] In this case, the program codes read from the storage medium can realize the functions of any of the above embodiments, and therefore the program codes and the storage medium storing the program codes constitute a part of the present application.

[0232] The storage medium for providing the program codes includes a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as a CD-ROM, a CD-R, a CD-RW, a DVD-ROM, a DVD-RYM, a DVD-RW, a DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program codes can be downloaded from a server computer via a communication network.

[0233] Furthermore, it should be understood that, not only the program codes read by the computer can be executed to complete the actual operation, but also the operating system and the like operating on the computer can be instructed based on the program codes to complete part or all of the actual operation, thereby realizing the function of any one of the above embodiments.

[0234] Furthermore, it should be understood that, not only the program codes read by the computer can be executed to complete the actual operation, but also the operating system and the like operating on the computer can be instructed based on the program codes to complete part or all of the actual operation, thereby realizing the function of any one of the above embodiments.

[0235] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A federated cross-modal hash retrieval method based on prototype privacy encryption, characterized in that, The method is specifically as follows: Constructing a data set: obtaining and organizing public data sets of image modalities and text modalities, dividing each public data set into a training set, a test set and a retrieval set, and combining to generate a final data set; Deep feature extraction: original data of public data sets of image modalities and text modalities are respectively subjected to feature extraction through a pre-trained model to obtain image modality deep features and text modality deep features, which are used to reflect the semantics of modality data; Constructing a federated cross-modality hash retrieval model: a federated cross-modality hash retrieval model based on prototype privacy encryption is constructed by using a neural network and a deep learning method; specifically as follows: Prototype privacy secure symmetric encryption: taking image modality deep features and text modality deep features as inputs, prototype information is generated in the process of federated learning to ensure that privacy information is not leaked in the process of information exchange and model parameter updating, and a decrypted global prototype is output; Personalized knowledge preservation: taking client modality initial parameters as inputs, updated parameters are obtained through an update super network part and a judgment super network extension part, a difference value is calculated according to the updated parameters and the input parameters, and finally the updated parameters are output; Hash retrieval query: a hash code generated by the client is used for retrieval query, and an average precision mean value is obtained as an evaluation index of retrieval performance; Training the model: the federated cross-modality hash retrieval model based on prototype privacy encryption is trained in the final data set; Wherein, the prototype privacy secure symmetric encryption is specifically as follows: Client local prototype generation: local prototypes of image modalities and text modalities in the client are respectively generated, which are used to capture information specific to the client, and the formula is as follows: where x i and y i denote the image modality deep feature and the text modality deep feature on the i-th client, respectively; m i denotes the number of samples on the i-th client; and denote the image modality hash function and the text modality hash function, respectively; and denote the image modality local prototype and the text modality local prototype in the i-th client, respectively; and are the training parameters. Client local prototype encryption: an encryption operation of the local prototype for identity authentication is constructed to ensure the security of the local prototype generated by the client, and the formula is as follows: where e(·) and ε(·) represent encryption operations; represents the prototype information of the i-th client; represents the noise encryption corresponding to the i-th client; Server-side local prototype decryption: a decryption operation of the local prototype for identity authentication is constructed on the server side to subsequently aggregate the local prototype safely, and the formula is as follows: where d(·) and D(·) represent decryption operations; represents client-local prototype encryption; auth represents server- versus-local-prototype authentication information; and χ represents a key. Server-side global prototype encryption: an encryption operation of the global prototype for identity authentication is constructed to ensure that the aggregated global prototype on the server side can be safely transmitted to the client, and the formula is as follows: where N denotes the number of clients; P gx and P gy represent global prototypes of image modalities and text modalities; and are client local prototypes, and the final global prototype of a client is represented as P g = {P gx , P gy}; Server-side global prototype encryption: an encryption operation of the global prototype for identity authentication is constructed to ensure that the aggregated global prototype on the server side can be safely transmitted to the client, and the formula is as follows: e(P g ) = e(P g + n g ); auth' = J(x, P g ); where e(·) and ε(·) denote encryption operations; P g denotes the final global prototype of the client; n g denotes the noise encryption on the server side; auth' denotes the authentication signal corresponding to the global prototype; Client global prototype decryption: after receiving the encrypted global prototype, the client global prototype decryption process is used for safe decryption to ensure that the client receives a safe and complete global prototype; the formula is as follows: d(P g ) = D(e(P g )-n g ) = P g ; where d(·) and D(·) denote decryption operations; e(P g ) denotes a global prototype of server-side encryption.

2. The federated cross-modal hash retrieval method based on prototype privacy-encryption according to claim 1, characterized in that, The construction of the data set is specifically as follows: Obtaining public data sets: obtaining and downloading public data sets containing image modalities and text modalities from public network resources as original data sets; wherein, the public data sets include FashionVC data sets and Ssense data sets; Dividing the downloaded original data set, specifically as follows: Randomly selecting any part of image-text pairs on the FashionVC data set and the Ssense data set as a test set for evaluating test performance; Excluding the FashionVC dataset and the test set image-text pair data in the Ssense dataset, randomly extracting any part of the remaining image-text pair as the retrieval dataset for retrieval operation; In the retrieval dataset, the remaining image-text pair of the dataset after removing the test set and retrieval set is used as the training set for the objective function training.

3. The federated cross-modal hash retrieval method based on prototype privacy-encryption according to claim 1, characterized in that, The deep feature extraction is as follows: Extracting image modal deep features: using a pre-trained model ViT as an image encoder to realize deep feature extraction of the image modal; the specific process is as follows: Preprocessing the original data of the image modal: uniformly crop and reshape the input image to HxW pixel size, and convert the reshaped image into a flattened two-dimensional plane image block; Pre-training ViT image encoder: feature extraction of the image modal is performed by stacking multiple Transformer layers, and global average pooling operation is performed on all position outputs of the last Transformer layer to obtain the global average pooling operation representation of the entire image; Extracting image features: performing linear projection on the average pooling representation output by the image modal to obtain the embedding feature representation of the image, which is a vector used to capture high-level semantic information of the image; Extracting text modal deep features: using a pre-trained model BERT as a text encoder to realize deep feature extraction of the text modal; the specific process is as follows: Preprocessing the original data of the text modal: cutting the input text into multiple segments, one word per segment, returning a list, and adding an identifier to the list to convert it into a vector representation; and considering the positional relationship of words in the text sentence; Pre-training BERT text encoder: through the multi-head self-attention mechanism network, different parts in the input sequence are focused on different positions to capture complex context relationships, and a feedforward neural network is applied after each attention layer to perform nonlinear transformation on the features; Extracting text features: applying layer normalization after the multi-head self-attention operation and feedforward neural network operation to obtain the text deep feature representation.

4. The federated cross-modal hash retrieval method based on prototype privacy encryption according to claim 1, characterized in that, The specific process of personalized knowledge preservation is as follows: update super network: after element-wise multiplication and the difference computation process updates the modal super network; wherein, represents the image modal super network; represents the text modal super network; and represents the super network embedding vector; and represents the super network parameters; and represents the image and text modal parameters, respectively; and represents the updated image and text modal parameters, respectively; Judging the super network expansion: using the super network expansion strategy to selectively retain and update the previously obtained category knowledge, and detecting whether the previously saved category knowledge is consistent with the newly uploaded category knowledge: If the category knowledge is consistent, the parameter range will not be expanded to merge new knowledge.

5. The federated cross-modal hash retrieval method based on prototype privacy-encryption according to claim 1, characterized in that, Hash retrieval query is a similarity retrieval by calculating the Hamming distance of the hash code, and the average precision is calculated multiple times for iteration to evaluate the retrieval performance, the specific process is as follows: For each modality, calculate the average precision under different recall rates, the formula is as follows: Where N represents the total number of samples associated with the retrieval sample; P(m) represents the precision of the first M instances related, when the mth sample is related to the retrieval sample, δ(m) = 1; otherwise, δ(m) = 0; The average precision mean is calculated according to the average precision, the formula is as follows: Where Q represents the number of retrieval sets; AP(i) represents the i-th average precision value; For each modality, calculate the average precision under different recall rates, and average the average precision of each modality to obtain the average precision mean of each modality.

6. The federated cross-modal hash search method based on prototype privacy encryption according to any one of claims 1 to 5, characterized in that, The specific process of training the model is as follows: Constructing loss function: the goal of the federated cross-modal hash retrieval model based on the prototype privacy encryption is to compare and match the hash code with other clients under the premise of encryption and privacy protection, to realize knowledge sharing and collaborative learning, so as to obtain high-quality hash code, and to optimize through loss function, and through the label L (i) , hash code B (i) , category similarity transformation matrix C (i) , and global prototype P g Variable optimization update; specific as follows: Minimize the optimization update B (i) The formula is as follows: wherein, represents an image hashing function; represents a text hashing function, for eliminating semantic gap and achieving better mapping from multi-modal data to hash codes, minimizing the difference between hash codes and the image hashing function and the text hashing function; Shared optimization update B (i) The formula is as follows: Transforming the objective function into the operational form of g helps to solve the statistical heterogeneity between clients; P g represents the final global prototype of the client, which helps to capture important information that may not exist alone in the local category relationship; Optimized update C (i) The formula is as follows: The target function is converted into The operation form of the target function is helpful to keep the similarity between the samples in the original data space and the corresponding hash codes, align the hash codes with the category similarity transformation matrix, ensure that the samples with labels or categories have similar hash codes, maintain the semantic structure and organization of the data; An optimization model; using the Adam algorithm as the optimization function of the model; wherein the learning rate parameter is set to 0.0001, and other hyperparameters can use the default values in PyTorch.

7. A federated cross-modal hash retrieval system based on prototype privacy-encryption, characterized in that, The system is used to implement the prototype privacy encryption-based federated cross-modal hash retrieval method according to any one of claims 1 to 6; the system comprises: A data set construction module is configured to obtain and organize public data sets of image modalities and text modalities, divide each public data set into a training set, a test set and a retrieval set, and combine to generate a final data set; A deep feature extraction module is configured to extract features from original data of the public data sets of the image modalities and the text modalities respectively by using a pre-trained model, to obtain image modality deep features and text modality deep features, and to use the image modality deep features and the text modality deep features to reflect the semantics of the modal data; A model construction module is configured to construct a prototype privacy encryption-based federated cross-modal hash retrieval model by using a neural network and a deep learning method; the model construction module comprises: A prototype privacy security symmetric encryption submodule is configured to use the image modality deep features and the text modality deep features as inputs to generate prototype information in a federated learning process, to ensure that privacy information is not leaked in the process of information exchange and model parameter updating, and to output decrypted global prototypes; specifically, the image modality deep features and the text modality deep features are used as inputs to generate local prototypes in a client local prototype generation part, and then the local prototypes are encrypted in a client local prototype encryption part, the encrypted local prototypes are uploaded to a server local prototype decryption part to decrypt the local prototypes, then a server global prototype generation part aggregates the decrypted local prototypes to generate a global prototype, then a server global prototype encryption part encrypts the global prototype, and finally a client global prototype decryption part decrypts the encrypted global prototype to output the decrypted global prototype. An individualized knowledge preservation submodule is configured to use client modality initial parameters as inputs, to obtain updated parameters through an update hypernetwork part and a judgment hypernetwork expansion part, to calculate a difference value based on the updated parameters and the input parameters, and to finally output the updated parameters; specifically, the client initial parameters are used as inputs to calculate a difference value between the updated parameters and the input parameters through the update hypernetwork part, then the judgment hypernetwork expansion part is used to effectively preserve previously obtained knowledge and to adapt to new data, to perform individualized adjustment based on specific tasks and data of each client, to maximize the performance and generalization of the model, and to alleviate the catastrophic forgetting of the hypernetwork individualized parameters, and finally to output the difference updated parameters. A hash retrieval query submodule is configured to perform retrieval query by using hash codes generated by the client, to obtain an average precision mean value as an evaluation index of retrieval performance; specifically, the hash codes generated by the client are used to perform similarity retrieval by calculating a Hamming distance, to calculate an average precision by multiple iterations, to then calculate an average precision mean value, and to finally obtain the average precision mean value to evaluate the retrieval performance. The training model module is configured to construct a loss function required in a training process of a prototype privacy-encrypted federated cross-modal hash retrieval model, and complete optimization training of the prototype privacy-encrypted federated cross-modal hash retrieval model.

8. An electronic device, comprising: The method comprises the following steps: a memory and at least one processor; wherein the memory has stored thereon a computer program; the at least one processor executes the computer program stored in the memory, so that the at least one processor executes the prototype privacy-encrypted federated cross-modal hash retrieval method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored therein a computer program, which can be executed by a processor to implement the prototype privacy-encrypted federated cross-modal hash retrieval method according to any one of claims 1 to 6.

Images