A personalized federated adversarial cross-modal hashing method based on dual prototypes

Abstract

The present invention discloses a personalized federated adversarial cross-modal hashing method based on dual prototypes, which mainly includes the following steps: at the server side, clustering the local class-level prototypes uploaded by the client to obtain clustered prototypes, averaging to obtain unbiased prototypes, and using a super network to generate personalized hierarchical weights of the client network according to gradient changes; the client uses the approximate hash codes and clustered prototypes of the two modal networks to construct an adversarial learning process, maintains the consistency between the unbiased prototype and the local class-level prototype, and also embeds the unbiased prototype into the training process of the client model; after the local training is completed, all network parameters and local class-level prototypes of the client are uploaded to the server; after the last communication round is completed, a hash code to be retrieved is generated based on the final client parameters, and the original data location information corresponding to the hash code with the smallest distance is used to provide the original data retrieval result.

Description

Technical Field

[0001] The present invention relates to the fields of federated learning and cross-modal hashing, and in particular to a personalized federated adversarial cross-modal hashing based on dual prototypes. Background Art

[0002] In the era of big data, massive amounts of data from various modalities, such as images, text, and audio, are experiencing explosive growth. Users are no longer satisfied with similarity retrieval within a single modality but are increasingly interested in cross-modal data retrieval—using data from one modality (e.g., text) to retrieve related data from another modality (e.g., images). Hashing retrieval is an approximate nearest neighbor retrieval method. Given a query, hashing aims to return the most visually or semantically similar images or text in the retrieval set. Due to its advantages in large-scale data retrieval speed and low storage consumption, it has been widely used in image retrieval, video retrieval, and cross-modal retrieval. However, training deep cross-modal hashing models requires a large amount of multimodal data, resulting in high transmission and maintenance costs. Due to restrictions imposed by laws and regulations on data security and privacy protection, collecting large amounts of multimodal data also poses significant risks of privacy breaches. Furthermore, businesses, schools, and government departments often possess their own "small data"—potentially valuable but unable to be directly shared—leading to a growing problem of data silos.

[0003] As a distributed machine learning method, federated learning enables collaborative learning to achieve a global model without exposing the data of participating parties, effectively preventing privacy leaks. Furthermore, this collaborative learning approach leverages the potential value of individual institutions' "small data," addressing the data silo problem to a certain extent. Recently, several cross-modal retrieval methods have been implemented within a federated framework. However, in real-world scenarios, data distribution varies across clients, resulting in heterogeneous data. Each client must independently design its own model, leading to model heterogeneity. Forcing each client to accept identical parameters prevents each client from optimally adapting to its local data. Furthermore, existing federated retrieval methods rely on aggregated local hashing models, meaning only local data is learned for similarity. Similarly, federated methods that use global prototypes tend to favor dominant client-class prototypes, similar to global models, resulting in suboptimal performance. Therefore, leveraging the interdependencies between multimodal data within a personalized federated framework to enable interaction between different users' data representations and knowledge, thereby improving computational and retrieval performance, remains a major challenge in federated learning. Summary of the Invention

[0004] In order to solve the problems of data heterogeneity, model heterogeneity and global prototype deviation in the current federated framework, the present invention provides a personalized federated adversarial cross-modal hashing method based on dual prototypes, which can perform cross-modal similarity retrieval under distributed storage conditions.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A personalized federated adversarial cross-modal hashing method based on dual prototypes is proposed. This method generates clustered prototypes and unbiased prototypes by clustering local class-level prototypes on the server. On the client, adversarial learning based on clustered prototypes and embedding strategies based on unbiased prototypes can solve data heterogeneity in federated learning while alleviating global prototype bias. In addition, a hypernetwork is used to dynamically update the weights of different layers of the client network to solve the model heterogeneity problem in federated learning and achieve customization of network parameters for different clients. The method includes the following steps:

[0007] Step 1) On the server side, the local class-level prototypes uploaded by the client are clustered to obtain cluster prototypes, which are then averaged to obtain unbiased prototypes. Furthermore, based on the gradient changes, four super-networks are used to generate hierarchical weights for all client image modality networks, image discriminators, and text modality networks, text discriminators. The server then transmits the updated hierarchical weights, cluster prototypes, and unbiased prototypes to the client.

[0008] In step 2), after receiving the hierarchical weights, the client updates the personalized model parameters of the image modality network, text modality network, and image and text discriminators. The client then extracts features from the private dataset through the image modality network and text modality network to obtain sample-rich semantic information and local client-level class-level prototypes. The adversarial learning process is constructed using the approximate hash codes and clustering prototypes of the two modality networks, allowing the client-side hash model to select a more suitable prototype to guide learning. The unbiased prototype is embedded in the training process of the client model and maintained in consistency with the local class-level prototype.

[0009] Step 3) After local training is completed, the client model parameters of the two modalities, the corresponding discriminator parameters, and the client's local class-level prototype will be uploaded to the server; continue the above process until the training stop condition is reached;

[0010] Step 4), after the last communication round is completed, the hash code to be retrieved is generated based on the final client parameters according to the data to be retrieved, and then the Hamming distance between the hash code to be retrieved and the hash code stored by the client is calculated. Then, the original data retrieval result is provided with the original data location information corresponding to the hash code with the smallest distance.

[0011] In particular, the cluster prototypes of the image modality and text modality in step 1) can be obtained by the following formula:

[0012]

[0013] in represents the local class-level prototype of the i-th client labeled c, m is the number of clients, J c The number of cluster prototypes k marked as c is the number of hash code bits. The cluster prototypes of image modality and text modality can be expressed as and The unbiased prototypes of the image modality and text modality in the method can be obtained by the following formula:

[0014]

[0015] Where C is the number of classes. Similarly, the image modality and the text modality have corresponding unbiased prototype matrices U x and U t .

[0016] In particular, in step 2), adversarial learning with cluster prototypes is constructed. In cross-modal retrieval, the corresponding adversarial loss functions for image and text modalities are:

[0017]

[0018] in is the i-th client image discriminator network, and Text Discriminator Network, J x and J t are the number of cluster prototypes for image modality and text modality respectively, Cluster prototype Combined with its corresponding label, a representative extended cluster prototype labeled c is generated. is the approximate hash code for the jth sample of the i-th client image modality The extended hash code obtained by combining with the corresponding label, is the extended clustering prototype of text modality, is the extended hash code of the text modality, and the total adversarial loss is expressed as:

[0019] In particular, in step 2), in order to reduce the impact of client class distribution on training and improve the retrieval performance of the local cross-modal hash model, the formula To maintain consistency between the client's local class-level prototype and the unbiased prototype, and are all images generated by the i-th client in the current communication round, the text class-level prototype matrix, and MSE(.) is the mean square error.

[0020] In particular, in step 2), the unbiased prototype is embedded into the training process of the local hash model, so that the local approximate hash code can interact with the unbiased prototype, and the deep hash model can simultaneously extract features and learn hash codes. For the i-th client, use represents the extracted image features, where Represents the parameters of the image modality network. For the text modality, use Or get the output of the text modality network, Representing the parameters of the text modality network, the embedding loss function of the unbiased prototype can be expressed as:

[0021]

[0022] in Y i ∈{-1,1} k×C are the label matrix of the i-th client, the hash code matrix of the sample, and the class hash code matrix; u i is the number of samples of the i-th client; α, β and μ are hyperparameters, so the loss function of the modal network is: Where η is a hyperparameter.

[0023] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0024] 1. Compared with the existing technology, the present invention proposes a personalized federated adversarial cross-modal hashing method based on dual prototypes. Compared with the centralized training that collects all image and text data, the present invention does not directly upload the data to the server, and can perform cross-modal similarity retrieval under distributed storage conditions, thus avoiding privacy leakage.

[0025] 2. Compared with the existing technology, the present invention uses a hypernetwork to dynamically update the weights of different layers of the hash model and the corresponding discriminator on the client, realizing personalized parameter customization for different clients. Since it does not use mathematical operations such as addition, it can solve the model heterogeneity problem in federated learning.

[0026] 3. Compared with the existing technology, the present invention constructs adversarial learning based on clustering prototypes, so that the client can select a prototype that adapts to its own data distribution for guidance and training. That is, clients with similar distributions will have similar feature representations, and the distinguishability of clients with different distributions can also be improved.

[0027] 4. Compared with the existing technology, the present invention embeds the unbiased prototype into the training process of the local hash model, which can obtain similarity information from other clients to enhance the performance of its own model. At the same time, maintaining the consistency of the unbiased prototype with the local class-level prototype can alleviate the global prototype deviation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the personalized federated adversarial cross-modal hashing method of the present invention;

[0029] Figure 2 is a network structure diagram of the personalized federated adversarial cross-modal hashing method of the present invention;

[0030] Figure 3 The discriminator structure details used by the present invention to construct adversarial learning; DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Figure 1 As shown, the personalized federated adversarial cross-modal hashing method based on dual prototypes described in this invention, compared with centralized training that collects all image and text data, does not directly transmit data to the server, but rather the network parameters, gradients, features and other values, which has a certain privacy protection effect. The solution includes the following steps:

[0032] See also Figure 2 First, in this embodiment, it is assumed that there are m clients in total. Without loss of generality, the model of this embodiment is established in the cross-modal retrieval task on the two modalities of image and text. The private cross-modal dataset of the i-th client can be expressed as n i is the number of samples owned by the i-th client, x j and t j The jth image (text) data of the i-th client, l j is the corresponding label vector.

[0033] Step 1), in this embodiment, cluster prototypes, unbiased prototypes, and hierarchical weights of the client are calculated on the server side. The FINCH unsupervised clustering method constructs an adjacency matrix through the nearest neighbor point (also called the first neighbor point) of each sample, and uses the nearest neighbor point as the basis for cluster selection. Such a parameter-free clustering method is suitable for federated learning with uncertain participant scale. Based on the assumption that prototypes from different distributions are unlikely to be first neighbor points, the FINCH unsupervised clustering method is used to select representative class prototypes rather than a single prototype. For all class-level prototypes uploaded by the client, the adjacency matrix of the c-th prototype can be defined as:

[0034]

[0035] in Indicates the prototype of the cth class from the i-th client The first adjacent point. Represents the local class-level prototype of the i-th client labeled c. For the i-th client, the image modality and the text modality should have corresponding class-level prototypes and The calculation is as follows:

[0036]

[0037] Based on the clustering results, the cluster prototypes can be obtained by averaging the class-level prototypes belonging to the same cluster. Such an operation helps to select representative class-level prototypes in the embedding space. Therefore, the cluster prototypes of image modality and text modality can be obtained by the following formula:

[0038]

[0039] Where m is the number of clients, J c The number of cluster prototypes is labeled c, k is the number of hash code bits,

[0040] in represents all private data of the i-th client marked as c, f i (.) and g i (.) represent the image modality and text modality networks of the i-th client respectively. Through the clustering algorithm, the clustering prototypes of the image modality and text modality can be obtained and The unbiased prototype is obtained by averaging the cluster prototypes belonging to the same class, as shown below:

[0041] Where C is the total number of classes. Similarly, both the image modality and the text modality should have corresponding unbiased prototype matrices U x and U t .

[0042] In addition, the super network updates the client network parameters layer by layer based on the network parameters and gradients of the current round. Because it does not use mathematical operations such as addition, it can solve the model heterogeneity problem in federated learning. According to the gradient changes, the four super networks are used to generate the layered weights of all client image modality networks, image discriminators, and text modality networks, and text discriminators. A super network contains four fully connected layers with ReLU linear activation functions. Then, the server will update the layered weights and abstract clustering prototypes. and the unbiased prototype U x , Ut After the update, it is passed to the client;

[0043] In step 2), after receiving the hierarchical weights, the client updates the personalized model parameters of the modal network and the discriminator; the client extracts the features of the private dataset through the image and text modal networks to obtain sample-rich semantic information and class-level prototypes of the local client.

[0044] Step 3), see Figure 3 In this embodiment, an adversarial learning strategy based on cluster prototypes ensures that clients with similar distributions have similar feature representations while improving the ability to distinguish clients with dissimilar distributions. Specifically, two fully connected layers with ReLU linear activation functions and a discriminator with a Sigmoid layer are used to construct adversarial learning with cluster prototypes. In this embodiment, the entire client-side hash model is equivalent to the generator in the adversarial network, and the cluster prototypes are equivalent to the real dataset. During training, the client-side hash model can acquire knowledge from other client data from the cluster prototypes, allowing the client-side hash model to select class-level prototypes that are more suitable for its own data distribution to guide learning. The output of the discriminator is a probability score between 0 and 1, with an output closer to 1 indicating that the input data is more likely to come from the cluster prototype. During the learning process, the approximate hash code generated by the client-side hash model will become increasingly similar to the cluster prototype until the client-side hash model can no longer benefit from the cluster prototype. Considering cross-modal retrieval, an adversarial learning process should also be established for the text modality. The network structure is consistent with the image modality, only the input is different. In addition, the corresponding adversarial loss functions for image and text modalities are:

[0045] in is the i-th client image discriminator network, and Text Discriminator Network, J x and J t are the number of cluster prototypes for image modality and text modality respectively, Cluster prototype Combined with its corresponding label y, a representative extended cluster prototype labeled c is generated. is the approximate hash code for the jth sample of the i-th client image modality With the corresponding label The extended hash code obtained after the combination, the labels here are all in one-hot encoding form. The combination of class labels supplements the input of the discriminator. Similarly, is the extended clustering prototype of text modality, The extended hash code for the text modality. The image discriminator parameters are updated using the backpropagation algorithm. and text discriminator parameters The total adversarial loss can be expressed as:

[0046] In step 4), the scales of cluster prototypes vary, so a fair and stable convergence point cannot be guaranteed. Compared with the global prototype, the unbiased prototype largely avoids the model from being biased towards the class with more samples and provides a fair and stable convergence target. In order to reduce the impact of client class distribution on training and improve the retrieval performance of the local cross-modal hash model, this embodiment uses the formula To maintain consistency between the client's local class-level prototype and the unbiased prototype, and are all images and text class-level prototype matrices generated by the i-th client in the current communication round, and are calculated using the following formula:

[0047]

[0048] in and are the class-level prototypes of the i-th client image modality and text modality, respectively, and MSE(.) is the mean square error.

[0049] Step 5), this embodiment embeds the unbiased prototype into the training process of the local hash model. For the local class-level prototype uploaded by each client, after the clustering average and other operations in step 1, an unbiased prototype that can represent the class information of all clients is obtained. This variable is used for similarity learning of local clients (also metric learning in hashing), so that each client can obtain the class-level similarity of other clients. Specifically, the deep hash model can simultaneously extract features and learn hash codes. For the i-th client, use represents the extracted image features, where Represents the parameters of the image modality network. Similarly, for the text modality, use Or get the output of the text modality network, represents the parameters of the text modality network. The embedding loss function of the unbiased prototype can be expressed as:

[0050]

[0051] in Y i ∈{-1,1} k×C are the label matrix of the i-th client, the hash code matrix of the sample, and the class hash code matrix; u i is the number of samples of the i-th client; α, β and μ are hyperparameters. The image modality network parameters are updated using the back propagation algorithm. Text modality network parameters Use an optimization strategy to discretely generate the class hash code Y bit by biti , the hash code is expressed as i =sign(F i +G i ) is calculated. Therefore, the loss function of the final modal network is: Where η is a hyperparameter.

[0052] Step 6), after the client local training is completed, the client model parameters of the two modalities, the corresponding discriminator parameters and the client local class-level prototype All will be uploaded to the server; continue the above process until the training stop condition is reached. The total loss function is: Where λ, η and ξ are hyperparameters.

[0053] Step 7) After the last communication round is completed, a hash code is generated based on the data to be retrieved and the final client parameters. The Hamming distance between the hash code to be retrieved and the hash code stored on the client is calculated. The original data retrieval result is then provided based on the original data location information corresponding to the hash code with the smallest distance. Specifically, the hash code to be retrieved is XORed with the hash code stored on the client to obtain the Hamming distance of the hash codes. The hash codes are then sorted based on the Hamming distance to obtain the hash codes with the smallest distance, i.e., the data closest to the data to be retrieved. Finally, the original data is returned based on the original data location information corresponding to the hash code.

[0054] The number of clients m is set to 10, the number of client hash model training rounds is set to E = 5, the batch size is 128, and the number of communication rounds is set to R = 50. The embedding layer and hidden dimension of the hypernetwork are both 32. The hyperparameters are set as follows: α = 0.5, β = 0.5, μ = 10, λ = 10, η = 10 -5 The learning rate of the modality network is 0.0001, and the learning rates of the hypernetwork and discriminator are set to 0.001.

[0055] The experimental results of the present invention are as follows:

[0056] The average of the highest mAP values ​​of one-third of the clients is randomly selected as the final result. A total of 10 clients are set in the experiment, and 'centralized' means the result of concentrating all data on a single device for training. The largest mAP value in the table is bolded. The mAP results on FashionVC and Ssense are shown in Table 1. Obviously, the method proposed in the present invention has achieved the best performance on both datasets compared with federated frameworks such as FedAvg, FedCMR, FedProx, FedProto, and PLFedCMH. In particular, in FashionVC, it has an average improvement of 1.4% over the second-ranked method under different hash code bits. Even in the Ssense dataset with good results, it has an average improvement of 0.2% over the second-ranked method under different hash code bits.

[0057] The above embodiments are intended only to illustrate the technical solutions of the present invention and should not be construed as limiting the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they may modify the technical solutions described in the above embodiments or replace some or all of the technical features therein with equivalents. Such modifications or replacements do not alter the essence of the corresponding technical solutions, which fall within the scope of the technical solutions of the embodiments of the present invention.

[0058] Table 1

[0059]

Claims

1. A personalized federated adversarial cross-modal hashing method based on dual prototypes. This method generates clustered prototypes and unbiased prototypes by clustering local class-level prototypes on the server. On the client, adversarial learning based on clustered prototypes and an embedding strategy based on unbiased prototypes can address data heterogeneity in federated learning while mitigating global prototype bias. Furthermore, a hypernetwork is used to dynamically update the weights of different layers of the client network to address model heterogeneity in federated learning and enable customization of network parameters for different clients. This method includes the following steps: In step 1), on the server side, the local class-level prototypes uploaded by the client are clustered to obtain clustered prototypes, which are then averaged to obtain unbiased prototypes. In addition, based on the gradient changes, four super-networks are used to generate the hierarchical weights of the image modality network, image discriminator, and text modality network, text discriminator for all clients. The server then passes the updated stratification weights, cluster prototypes, and unbiased prototypes to the client; Step 2) After receiving the hierarchical weights, the client updates the personalized model parameters of the image modality network, text modality network, image discriminator, and text discriminator; The client then extracts features from the private dataset through the image modality network and the text modality network to obtain sample-rich semantic information and class-level prototypes of the local client; We use the approximate hash codes and clustering prototypes of the two modal networks to construct an adversarial learning process, enabling the client-side hash model to select a more suitable prototype to guide learning. We also embed the unbiased prototype into the training process of the client model and maintain its consistency with the local class-level prototype. Step 3), after local training is completed, the client model parameters of both modalities, as well as the corresponding discriminator parameters and the client's local class-level prototypes will be uploaded to the server; Continue to cycle the above process until the training stop condition is reached; Step 4), after the last communication round is completed, the hash code to be retrieved is generated based on the final client parameters according to the data to be retrieved, and then the Hamming distance between the hash code to be retrieved and the hash code stored by the client is calculated. Then, the original data retrieval result is provided with the original data location information corresponding to the hash code with the smallest distance.

2. The dual-prototype-based personalized federated adversarial cross-modal hashing method according to claim 1, characterized in that: The cluster prototypes of the image modality and text modality in step 1) can be obtained by the following formula: in represents the local class-level prototype of the i-th client labeled c, m is the number of clients, J c The number of cluster prototypes k marked as c is the number of hash code bits. The cluster prototypes of image modality and text modality can be expressed as and The unbiased prototypes of the image modality and text modality in the method can be obtained by the following formula: Where C is the number of classes. Similarly, the image modality and the text modality have corresponding unbiased prototype matrices U x and U t .

3. The dual-prototype-based personalized federated adversarial cross-modal hashing method according to claim 1, characterized in that: In step 2), adversarial learning with cluster prototypes is constructed. In cross-modal retrieval, the corresponding adversarial loss functions for image and text modalities are: in is the i-th client image discriminator network, and Text Discriminator Network, J x and J t are the number of cluster prototypes for image modality and text modality respectively, Cluster prototype Combined with its corresponding label, a representative extended cluster prototype labeled c is generated. is the approximate hash code for the jth sample of the i-th client image modality The extended hash code obtained by combining with the corresponding label, is the extended clustering prototype of text modality, is the extended hash code of the text modality, and the total adversarial loss is expressed as:

4. The dual-prototype-based personalized federated adversarial cross-modal hashing method according to claim 1, characterized in that: In step 2), in order to reduce the impact of client class distribution on training and improve the retrieval performance of the local cross-modal hash model, the formula To maintain consistency between the client's local class-level prototype and the unbiased prototype, and are all images generated by the i-th client in the current communication round, the text class-level prototype matrix, and MSE(.) is the mean square error.

5. The dual-prototype-based personalized federated adversarial cross-modal hashing method according to claim 1, characterized in that: In step 2), the unbiased prototype is embedded into the training process of the local hash model, so that the local approximate hash code can interact with the unbiased prototype. The deep hash model can simultaneously extract features and learn hash codes. For the i-th client, use represents the extracted image features, where Represents the parameters of the image modality network. For the text modality, use Or get the output of the text modality network, Representing the parameters of the text modality network, the embedding loss function of the unbiased prototype can be expressed as: in Y i ∈{-1,1} k×C are the label matrix of the i-th client, the hash code matrix of the sample, and the class hash code matrix; u i is the number of samples of the i-th client; α, β and μ are hyperparameters, so the loss function of the modal network is: min L2+ηL3, where η is a hyperparameter.

Images