Personalized federated confrontation cross-modal hashing method based on double prototypes

Abstract

The invention discloses a personalized federal confrontation cross-modal hashing method based on double prototypes, which mainly comprises the following steps of: clustering local class-level prototypes uploaded by a client at a server side to obtain a clustering prototype, averaging to obtain an unbiased prototype, and according to gradient change, obtaining an unbiased prototype; generating personalized hierarchical weights for the client network using the super network; the client constructs an adversarial learning process by using approximate hash codes and clustering prototypes of the two modal networks, keeps the consistency between an unbiased prototype and a local class-level prototype, and further embeds the unbiased prototype into a training process of a client model; after local training is completed, all network parameters and local class-level prototypes of the client side are uploaded to the server; and after the last communication round is completed, generating a to-be-retrieved hash code based on the final client parameter, and providing an original data retrieval result according to the original data position information corresponding to the hash code with the minimum distance.

Description

Technical Field

[0001] The present invention relates to the fields of federated learning and cross-modal hashing, and in particular to a personalized federated adversarial cross-modal hashing based on dual prototypes. Background Art

[0002] In the era of big data, massive amounts of data in different modalities, such as images, texts, and audios, are growing explosively. Users are no longer satisfied with similarity retrieval between single-modal data, but are more concerned with cross-modal data retrieval, that is, using data from one modality (such as text modality) to retrieve related data from another modality (such as image modality). The hash retrieval method is an approximate nearest neighbor retrieval method. For a given query sample, the purpose of hash retrieval is to return the most visually or semantically similar image or text in the retrieval set. Because of its advantages of fast retrieval speed and low storage consumption for large-scale data, it has been widely used in image retrieval, video retrieval, and cross-modal retrieval. However, the training of deep cross-modal hash models requires a large amount of multimodal data, which will bring high transmission consumption and maintenance costs; restricted by laws and regulations related to data security and privacy protection, collecting a large amount of multimodal data will also lead to huge privacy leakage risks. In addition, enterprises, schools, or government departments all have their own "small data" that has potential value but cannot be directly shared, making the data island problem increasingly serious.

[0003] As a distributed machine learning method, federated learning can obtain a global model through collaborative learning without exposing the data of the participants, thus effectively avoiding privacy leakage. In addition, the potential value of "small data" of each institution is also reflected through this collaborative learning, so that the problem of data islands is also solved to a certain extent. Recently, some cross-modal retrieval methods have been implemented in the federated framework. However, in real scenarios, the data distribution of each client is usually different, resulting in data heterogeneity; each client needs to independently design its own model, resulting in model heterogeneity, and the strategy of forcing each client to accept the same parameters will prevent each client from better adapting to its own local data. In addition, the existing federated retrieval methods rely on aggregated local hash models, which means that only local data is learned for similarity; for federated methods using global prototypes, similar to global models, global prototypes will tend to favor the dominant customer class prototype, resulting in suboptimal performance. Therefore, how to use the interdependence between multimodal data in a personalized federated framework to enable data representation and knowledge of different users to interact and improve computing and retrieval performance is still a major problem in current federated learning. Summary of the invention

[0004] In order to solve the problems of data heterogeneity, model heterogeneity and global prototype deviation in the current federated framework, the present invention provides a personalized federated adversarial cross-modal hashing method based on dual prototypes, which can perform cross-modal similarity retrieval under distributed storage conditions.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A personalized federated adversarial cross-modal hashing method based on dual prototypes is proposed. The method generates clustered prototypes and unbiased prototypes by clustering local class-level prototypes on the server side. On the client side, adversarial learning based on clustered prototypes and embedding strategies of unbiased prototypes can solve data heterogeneity in federated learning while alleviating global prototype bias. In addition, the hypernetwork is used to dynamically update the weights of different layers of the client network to solve the model heterogeneity problem in federated learning and realize the customization of network parameters of different clients. The method includes the following steps:

[0007] Step 1), on the server side, cluster the local class-level prototypes uploaded by the client to obtain cluster prototypes, and then further obtain unbiased prototypes after averaging; in addition, according to the gradient change, four super networks are used to generate the hierarchical weights of all client image modality networks, image discriminators, and text modality networks, and text discriminators; then, the server side updates and transmits the updated hierarchical weights, cluster prototypes and unbiased prototypes to the client;

[0008] Step 2), after receiving the hierarchical weights, the client updates the personalized model parameters of the image modality network, text modality network, image discriminator, and text discriminator; then the client extracts the features of the private dataset through the image modality network and text modality network to obtain sample-rich semantic information and the class-level prototype of the local client; the adversarial learning process is constructed using the approximate hash codes and clustering prototypes of the two modality networks, so that the client hash model selects a more suitable prototype to guide learning, embeds the unbiased prototype into the training process of the client model, and maintains its consistency with the local class-level prototype;

[0009] Step 3), after the local training is completed, the client model parameters of the two modalities, the corresponding discriminator parameters and the client's local class-level prototype will be uploaded to the server; continue to cycle the above process until the training stop condition is reached;

[0010] Step 4), after the last communication round is completed, the hash code to be retrieved is generated based on the data to be retrieved and the final client parameters, and then the Hamming distance between the hash code to be retrieved and the hash code stored by the client is calculated, and then the original data retrieval result is provided with the original data location information corresponding to the hash code with the smallest distance.

[0011] In particular, the clustering prototypes of the image modality and the text modality in step 1) can be obtained by the following formula:

[0012]

[0013] in represents the local class-level prototype of the i-th client labeled c, m is the number of clients, J c The number of cluster prototypes marked as c is k, which is the number of hash code bits. The cluster prototypes of image modality and text modality can be expressed as and The unbiased prototypes of the image modality and text modality in the method can be obtained by the following formula:

[0014]

[0015] Where C is the number of classes. Similarly, the image modality and the text modality have corresponding unbiased prototype matrices U x and U t .

[0016] In particular, in step 2), adversarial learning with clustering prototypes is constructed. In cross-modal retrieval, the adversarial loss functions corresponding to the image and text modalities are respectively:

[0017]

[0018] in is the i-th client image discriminator network, and Text Discriminator Network, J x and J t are the number of cluster prototypes for image modality and text modality respectively, Cluster prototype Combined with its corresponding label, a representative extended cluster prototype labeled c is generated. is the approximate hash code for the jth sample of the i-th client image modality The extended hash code obtained by combining with the corresponding label, is the extended clustering prototype of text modality, is the extended hash code of the text modality, and the total adversarial loss is expressed as:

[0019] In particular, in step 2), in order to reduce the impact of client class distribution on training and improve the retrieval performance of the local cross-modal hash model, the formula To maintain the consistency between the client's local class-level prototype and the unbiased prototype, and are all images generated by the i-th client in the current communication round, the text class-level prototype matrix, and MSE(.) is the mean square error.

[0020] In particular, in step 2), the unbiased prototype is embedded into the training process of the local hash model, so that the local approximate hash code can interact with the unbiased prototype, and the deep hash model can simultaneously extract features and learn hash codes. For the i-th client, use represents the extracted image features, where Represents the parameters of the image modality network. For the text modality, use Or the output of the text modality network, Representing the parameters of the text modality network, the embedding loss function of the unbiased prototype can be expressed as:

[0021]

[0022] in Y i ∈{-1,1} k×C are the label matrix of the i-th client, the hash code matrix of the sample, and the class hash code matrix; u i is the number of samples of the i-th client; α, β and μ are hyperparameters, so the loss function of the modal network is: Where η is a hyperparameter.

[0023] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0024] 1. Compared with the prior art, the present invention proposes a personalized federated adversarial cross-modal hashing method based on dual prototypes. Compared with the centralized training that collects all image text data, the present invention does not directly upload the data to the server, and can perform cross-modal similarity retrieval under distributed storage conditions to avoid privacy leakage.

[0025] 2. Compared with the prior art, the present invention uses a hypernetwork to dynamically update the weights of different layers of the hash model and the corresponding discriminator on the client, thereby realizing personalized parameter customization for different clients. Since it does not use mathematical operations such as addition, it can solve the model heterogeneity problem in federated learning.

[0026] 3. Compared with the prior art, the present invention constructs adversarial learning based on clustering prototypes, so that the client can select a prototype that adapts to its own data distribution for guidance and training, that is, clients with similar distributions will have similar feature representations, and the distinguishability of clients with different distributions can also be improved.

[0027] 4. Compared with the prior art, the present invention embeds the unbiased prototype into the training process of the local hash model, and can obtain similarity information from other clients to enhance the performance of its own model. At the same time, maintaining the consistency of the unbiased prototype with the local class-level prototype can alleviate the global prototype deviation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the personalized federated adversarial cross-modal hashing method of the present invention;

[0029] Figure 2 It is a network structure diagram of the personalized federated adversarial cross-modal hashing method of the present invention;

[0030] Figure 3 It is the structural details of the discriminator used by the present invention to construct adversarial learning; DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Figure 1 As shown, the personalized federated adversarial cross-modal hashing method based on dual prototypes described in the present invention, compared with the centralized training that collects all image text data, federated learning does not directly transmit data to the server, but network parameters, gradients, features and other values, which has a certain effect of protecting privacy. The scheme includes the following steps:

[0032] See also Figure 2 First, in this embodiment, it is assumed that there are m clients in total. Without loss of generality, the model of this embodiment is established in the cross-modal retrieval task on the two modalities of image and text. The private cross-modal dataset of the i-th client can be expressed as n i is the number of samples owned by the i-th client, x j and t j The jth image (text) data of the i-th client, l j is the corresponding label vector.

[0033] Step 1), in this embodiment, clustering prototypes, unbiased prototypes, and hierarchical weights of the client are calculated on the server side. The FINCH unsupervised clustering method constructs an adjacency matrix through the nearest neighbor point (also called the first neighbor point) of each sample, and uses the nearest neighbor point as the basis for cluster selection. Such a parameter-free clustering method is suitable for federated learning with uncertain participant scales. Based on the assumption that prototypes from different distributions are unlikely to be first neighbor points, the FINCH unsupervised clustering method selects representative class prototypes rather than a single prototype. For all class-level prototypes uploaded by the client, the adjacency matrix of the c-th prototype can be defined as:

[0034]

[0035] in Indicates the prototype of the cth class from the i-th client The first adjacent point of . Represents the local class-level prototype of the i-th client labeled c. For the i-th client, the image modality and the text modality should have corresponding class-level prototypes and The calculation is as follows:

[0036]

[0037] Based on the clustering results, the cluster prototypes can be obtained by averaging the class-level prototypes belonging to the same cluster. Such operations help to select representative class-level prototypes in the embedding space. Therefore, the cluster prototypes of image modality and text modality can be obtained by the following formula:

[0038]

[0039] Where m is the number of clients, J c The number of cluster prototypes is labeled c, k is the number of hash code bits,

[0040] in represents all private data marked as c for the i-th client, f i (.) and g i (.) represent the image modality and text modality networks of the i-th client respectively. Through the clustering algorithm, the clustering prototypes of the image modality and text modality can be obtained and The unbiased prototype is obtained by averaging the cluster prototypes belonging to the same class, as shown below:

[0041] Where C is the total number of classes. Similarly, both the image modality and the text modality should have corresponding unbiased prototype matrices U x and U t .

[0042] In addition, the hypernetwork updates the client network parameters layer by layer according to the network parameters and gradients of the current round. Because it does not use mathematical operations such as addition, it can solve the model heterogeneity problem in federated learning. According to the gradient changes, the four hypernetworks are used to generate the hierarchical weights of all client image modality networks, image discriminators, and text modality networks, and text discriminators. A hypernetwork contains four fully connected layers with ReLU linear activation functions. Then, the server will update the hierarchical weights and abstract clustering prototypes. and the unbiased prototype U x , Ut The update is passed to the client;

[0043] Step 2), after receiving the hierarchical weights, the client updates the personalized model parameters of the modal network and the discriminator; the client extracts the features of the private dataset through the image and text modal networks to obtain sample-rich semantic information and class-level prototypes of the local client.

[0044] Step 3), see Figure 3 In this embodiment, the adversarial learning strategy based on clustering prototypes enables clients with similar distributions to have similar feature representations, while improving the distinction between clients with dissimilar distributions. Specifically, two fully connected layers with ReLU linear activation functions and a Sigmoid layer discriminator are used to construct adversarial learning with clustering prototypes. In this embodiment, the entire client hash model is equivalent to the generator in the adversarial network, and the clustering prototype is equivalent to the real data set. During the training process, the client hash model can obtain knowledge from other client data from the clustering prototype, so that the client hash model can select a class-level prototype that is more suitable for its own data distribution to guide learning. The output of the discriminator is a probability score between 0 and 1, and an output closer to 1 indicates that the input data is more likely to come from the clustering prototype. During the learning process, the approximate hash code generated by the client hash model will become more and more similar to the clustering prototype until the client hash model can no longer benefit from the clustering prototype. Considering that it is a cross-modal retrieval, the text modality should also establish an adversarial learning process. The network structure is consistent with the image modality, but the input is different. In addition, the corresponding adversarial loss functions for image and text modalities are:

[0045] in is the i-th client image discriminator network, and Text Discriminator Network, J x and J t are the number of cluster prototypes for image modality and text modality respectively, Cluster prototype Combined with its corresponding label y, a representative extended cluster prototype labeled c is generated. is the approximate hash code for the jth sample of the i-th client image modality With corresponding tags The extended hash code obtained after the combination, the labels here are all in one-hot encoding form. The combination of class labels supplements the input of the discriminator. Similarly, is the extended clustering prototype of text modality, The extended hash code for the text modality. The image discriminator parameters are updated using the back-propagation algorithm. and the text discriminator parameters The total adversarial loss can be expressed as:

[0046] In step 4), the scales of cluster prototypes are different, so a fair and stable convergence point cannot be guaranteed. Compared with the global prototype, the unbiased prototype largely avoids the model from being biased towards the class with more samples and provides a fair and stable convergence target. In order to reduce the impact of client class distribution on training and improve the retrieval performance of the local cross-modal hash model, this embodiment uses the formula To maintain the consistency between the client's local class-level prototype and the unbiased prototype, and are all images and text class-level prototype matrices generated by the i-th client in the current communication round, and are calculated using the following formula:

[0047]

[0048] in and are the class-level prototypes of the i-th client image modality and text modality respectively, and MSE(.) is the mean square error.

[0049] Step 5), this embodiment embeds the unbiased prototype into the training process of the local hash model. For each local class-level prototype uploaded by each client, after the clustering average and other operations in step 1, an unbiased prototype that can represent the class information of all clients is obtained. This variable is used for similarity learning of local clients (also metric learning in hashing), so that each client can obtain the class-level similarity of other clients. Specifically, the deep hash model can extract features and learn hash codes at the same time. For the i-th client, use represents the extracted image features, where Represents the parameters of the image modality network. Similarly, for the text modality, use Or the output of the text modality network, represents the parameters of the text modality network. The embedding loss function of the unbiased prototype can be expressed as:

[0050]

[0051] in Y i ∈{-1,1} k×C are the label matrix of the i-th client, the hash code matrix of the sample, and the class hash code matrix; u i is the number of samples of the i-th client; α, β and μ are hyperparameters. Use the back propagation algorithm to update the image modality network parameters Text modality network parameters Use an optimization strategy to generate the class hash code Y discretely bit by biti , the hash code is expressed by i =sign(F i +G i ) is calculated. Therefore, the loss function of the final modal network is: Where η is a hyperparameter.

[0052] Step 6), after the client local training is completed, the client model parameters of the two modalities, the corresponding discriminator parameters and the client local class-level prototype All will be uploaded to the server; continue to cycle the above process until the training stop condition is reached. The total loss function is: Among them, λ, η and ξ are hyperparameters.

[0053] Step 7), after the last communication round is completed, the hash code to be retrieved is generated based on the data to be retrieved and the final client parameters, and then the Hamming distance between the hash code to be retrieved and the hash code stored by the client is calculated, and then the original data retrieval result is provided based on the original data location information corresponding to the hash code with the smallest distance. That is, the hash code to be retrieved and the hash code stored by the client are XORed to obtain the Hamming distance of the hash code, and the hash codes are sorted according to the Hamming distance to obtain several hash codes with the smallest distance, that is, the corresponding data closest to the data to be retrieved, and finally, several pieces of original data are returned according to the original data location information corresponding to the hash code.

[0054] The number of clients m=10, the number of client hash model training rounds E=5, the batch size is 128, and the number of communication rounds R=50. The embedding layer and hidden dimension of the hypernetwork are both 32. The hyperparameters are set as follows: α=0.5, β=0.5, μ=10, λ=10, η=10 -5 The learning rate of the modality network is 0.0001, and the learning rates of the hypernetwork and the discriminator are set to 0.001.

[0055] The experimental results of the present invention are as follows:

[0056] The average of the highest mAP values ​​of one-third of the clients is randomly selected as the final result. A total of 10 clients are set in the experiment, and 'centralized' means the result of concentrating all data on a single device for training. The largest mAP value in the table is bold. The mAP results on FashionVC and Ssense are shown in Table 1. Obviously, the method proposed in the present invention has achieved the best performance on both datasets compared with federated frameworks such as FedAvg, FedCMR, FedProx, FedProto, and PLFedCMH, especially in FashionVC, which is an average improvement of 1.4% over the second-ranked method at different numbers of hash code bits. Even in the Ssense dataset with a good effect, it is an average improvement of 0.2% over the second-ranked method at different numbers of hash code bits.

[0057] The above embodiments are only used to illustrate the technical solutions of the present invention and should not be regarded as limiting the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or perform equivalent replacements on some or all of the technical features therein. These modifications or replacements do not change the essence that the corresponding technical solutions belong to the technical solutions of the embodiments of the present invention.

[0058] Table 1

[0059]

Claims

1. A personalized federated adversarial cross-modal hashing method based on dual prototypes. The method generates clustered prototypes and unbiased prototypes by clustering local class-level prototypes on the server side. On the client side, adversarial learning based on clustered prototypes and embedding strategies of unbiased prototypes can solve data heterogeneity in federated learning while alleviating global prototype bias. In addition, the hypernetwork is used to dynamically update the weights of different layers of the client network to solve the model heterogeneity problem in federated learning and realize the customization of network parameters of different clients. The method includes the following steps: Step 1), on the server side, cluster the local class-level prototypes uploaded by the client to obtain clustered prototypes, and then further obtain unbiased prototypes after averaging; in addition, according to the gradient change, four super networks are used to generate the hierarchical weights of all client image modality networks, image discriminators, and text modality networks and text discriminators; Then, the server passes the updated stratification weights as well as the clustering prototype and the unbiased prototype to the client; Step 2), after receiving the hierarchical weights, the client updates the personalized model parameters of the image modality network, the text modality network, the image discriminator, and the text discriminator; The client then extracts features of the private dataset through the image modality network and the text modality network to obtain sample-rich semantic information and class-level prototypes of the local client; The adversarial learning process is constructed using the approximate hash codes and clustering prototypes of the two modal networks, so that the client hash model selects a more suitable prototype to guide learning, embeds the unbiased prototype into the training process of the client model, and maintains its consistency with the local class-level prototype; Step 3), after the local training is completed, the client model parameters of both modalities, as well as the corresponding discriminator parameters and the client's local class-level prototypes will be uploaded to the server; Continue to cycle the above process until the training stop condition is reached; Step 4), after the last communication round is completed, the hash code to be retrieved is generated based on the data to be retrieved and the final client parameters, and then the Hamming distance between the hash code to be retrieved and the hash code stored by the client is calculated, and then the original data retrieval result is provided with the original data location information corresponding to the hash code with the smallest distance.

2. The dual-prototype-based personalized federated adversarial cross-modal hashing method according to claim 1, characterized in that: The clustering prototypes of the image modality and text modality in step 1) can be obtained by the following formula: in represents the local class-level prototype of the i-th client labeled c, m is the number of clients, J c The number of cluster prototypes marked as c is k, which is the number of hash code bits. The cluster prototypes of image modality and text modality can be expressed as and The unbiased prototypes of the image modality and text modality in the method can be obtained by the following formula: Where C is the number of classes. Similarly, the image modality and the text modality have corresponding unbiased prototype matrices U x and U t .

3. The dual-prototype-based personalized federated adversarial cross-modal hashing method according to claim 1, characterized in that: In step 2), adversarial learning with clustering prototypes is constructed. In cross-modal retrieval, the adversarial loss functions corresponding to the image and text modalities are: in is the i-th client image discriminator network, and Text Discriminator Network, J x and J t are the number of cluster prototypes for image modality and text modality respectively, Cluster prototype Combined with its corresponding label, a representative extended cluster prototype labeled c is generated. is the approximate hash code for the jth sample of the i-th client image modality The extended hash code obtained by combining with the corresponding label, is the extended clustering prototype of text modality, is the extended hash code of the text modality, and the total adversarial loss is expressed as:

4. The dual-prototype-based personalized federated adversarial cross-modal hashing method according to claim 1, characterized in that: In step 2), in order to reduce the impact of client class distribution on training and improve the retrieval performance of the local cross-modal hash model, the formula To maintain the consistency between the client's local class-level prototype and the unbiased prototype, and are all images generated by the i-th client in the current communication round, the text class-level prototype matrix, and MSE(.) is the mean square error.

5. The dual-prototype-based personalized federated adversarial cross-modal hashing method according to claim 1, characterized in that: In step 2), the unbiased prototype is embedded into the training process of the local hash model, so that the local approximate hash code can interact with the unbiased prototype. The deep hash model can simultaneously extract features and learn hash codes. For the i-th client, use represents the extracted image features, where Represents the parameters of the image modality network. For the text modality, use Or the output of the text modality network, Representing the parameters of the text modality network, the embedding loss function of the unbiased prototype can be expressed as: in Y i ∈{-1,1} k×C are the label matrix of the i-th client, the hash code matrix of the sample, and the class hash code matrix; u i is the number of samples of the i-th client; α, β and μ are hyperparameters, so the loss function of the modal network is: min L2+ηL3, where η is a hyperparameter.

Images