Cross-modal retrieval model training method, unsupervised federated learning system, and program product

Abstract

The application provides a cross-modal retrieval model training method, an unsupervised federated learning system and a program product. The method comprises: performing unsupervised training on a local generative adversarial network by taking cross-modal splicing features corresponding to local multi-modal data and local prototypes as constraints to obtain a local generator for generating local enhanced data; sending the local prototypes, the local generator and a local cross-modal retrieval model trained locally based on feature vectors of various types of modalities corresponding to the local multi-modal data to a server, so that the server obtains global training data according to the local prototypes and the local generator and trains a global cross-modal retrieval model corresponding to each local cross-modal retrieval model. The application can fully utilize the multi-modal relationship of local data in the process of training the global cross-modal retrieval model based on network learning and does not need to rely on high-quality data labeling, so as to improve the efficiency and reliability of the cross-modal retrieval model training.

Description

Technical Field

[0001] This application relates to the field of federated learning technology, and in particular to cross-modal retrieval model training methods, unsupervised federated learning systems and program products. Background Technology

[0002] In modern society, the booming development of multimodal data, such as images, text, and videos on the internet, makes robust cross-modal retrieval models a critical need. Among various advanced technologies, deep hashing techniques, utilizing deep neural networks, have significantly improved the accuracy and efficiency of large-scale cross-modal retrieval. However, training deep cross-modal hashing models still faces data privacy challenges. Real-world data is typically discretely distributed across multiple clients, accessible only to those clients, which include mobile phones, personal computers, and IoT devices. To meet privacy requirements, federated learning has been introduced into cross-modal retrieval. This allows servers to obtain a relatively good performance model without requiring the original client data. However, it still faces the challenge of global model bias caused by non-independent and identically distributed data: the quantity and category of data vary from client to client.

[0003] Currently, FedCAFE, a joint cross-modal hashing method with adaptive feature enhancement, is a supervised cross-modal retrieval method that introduces a novel adaptive feature enhancement module and a new weighted aggregation strategy. It fully utilizes the rich global information carried in the global model to constrain the model during local training. Meanwhile, PT-FUCH, a federated unsupervised cross-modal hashing method based on prototype transfer, protects local private data by exploring a unified global prototype across different clients without relying on any annotations. The global prototype guides local cross-modal hashing learning and promotes alignment of the feature space, thereby mitigating model bias caused by differences in the distribution of local multimodal data. Furthermore, it incorporates an adaptive cross-modal knowledge distillation method, transferring valuable semantic knowledge from the global model of a specific modality to the local prototype learning process, reducing the risk of overfitting.

[0004] However, the FedCAFE method heavily relies on high-quality labeled data, and in real-world scenarios, it is difficult to obtain sufficient high-quality labeled multimodal data, resulting in low feasibility of the method in practice. Furthermore, the guidance of the global prototype in PT-FUCH can only alleviate the model bias problem caused by non-independent and identically distributed data to a certain extent; the multimodal relationships of real data in various clients are not fully utilized. Summary of the Invention

[0005] In view of this, embodiments of this application provide a cross-modal retrieval model training method, an unsupervised federated learning system, and program products to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of this application provides a method for training a cross-modal retrieval model, comprising:

[0007] Using the cross-modal splicing features corresponding to local multimodal data and the local prototype obtained by clustering the cross-modal splicing features in advance as constraints, the local adversarial generative network is trained in unsupervised training so as to train the adversarial generative network into a local generator for generating local augmented data.

[0008] The local prototypes, local generators, and local cross-modal retrieval models trained locally based on feature vectors of various modalities corresponding to local multimodal data are all sent to the server. This allows the server to obtain global training data based on the received local prototypes and local generators, and to train a global cross-modal retrieval model corresponding to each local cross-modal retrieval model based on the global training data.

[0009] In some embodiments of this application, before performing unsupervised training on the local generative adversarial network using the cross-modal splicing features corresponding to the local multimodal data and the local prototype obtained by pre-clustering the cross-modal splicing features as constraints, the method further includes:

[0010] The local multimodal data is divided into training data pairs, wherein each training data pair contains first modal data and second modal data belonging to different modal types respectively;

[0011] The first modal data in each training data pair is input into the feature extraction model corresponding to the first modality, and the second modal data in each training data pair is input into the feature extraction model corresponding to the second modality, so that the feature extraction model corresponding to the first modality outputs the first modal feature vector corresponding to the first modal data, and the feature extraction model corresponding to the second modality outputs the second modal feature vector corresponding to the second modal data.

[0012] The training data is used to train a local cross-modal retrieval model on the corresponding first modality feature vector, second modality data, and hash loss, so that the local cross-modal retrieval model can be used to obtain the hash representations corresponding to the first modality data and the second modality data respectively;

[0013] Furthermore, each training data is projected onto and concatenated with its corresponding first modality feature vector and second modality data to obtain the cross-modal concatenated features corresponding to each training data.

[0014] Clustering and projection are performed on the cross-modal splicing features corresponding to each of the training data to obtain the local prototype corresponding to the local multimodal data.

[0015] In some embodiments of this application, the step of unsupervised training of a local generative adversarial network (GAN) using cross-modal splicing features corresponding to local multimodal data and local prototypes obtained by pre-clustering the cross-modal splicing features as constraints, to train the GAN into a local generator for generating local augmented data, includes:

[0016] The local adversarial generative network is iteratively trained in multiple rounds using cross-modal splicing features corresponding to local multimodal data, and preset training steps are executed in each round.

[0017] The training steps include:

[0018] If the current iteration is the first iteration, the cross-modal splicing features corresponding to the local multimodal data are input into the local adversarial generative network (GCN) so that the GCN outputs the local augmented data for the current iteration. If the current iteration is not the first iteration, the local GCN is adjusted based on the training loss obtained from the previous iteration, and the cross-modal splicing features corresponding to the local multimodal data are input into the adjusted GCN so that the GCN outputs the local augmented data for the current iteration.

[0019] Assign pseudo-labels to the local augmented data based on the local prototype;

[0020] Furthermore, the local augmentation data is divided into first-mode augmentation features and second-mode augmentation features according to modal type;

[0021] Based on the first modality enhancement features and the second modality enhancement features corresponding to the local augmented data, and the local prototype with the same pseudo-label as the local augmented data, the training loss with data quality constraints corresponding to the adversarial generative network in the next round is obtained.

[0022] In some embodiments of this application, the training loss with data quality constraints is solved based on the training loss function of the adversarial generative network;

[0023] The training loss function of the adversarial generative network includes: a generator training loss function and a discriminator training loss function;

[0024] The generator training loss function is solved based on the client loss and a preset generator loss; the discriminator training loss function is solved based on the client loss and a preset discriminator loss.

[0025] The client loss is calculated based on intra-class and inter-class contrast alignment loss, distribution alignment loss, and hash loss used to train the local cross-modal retrieval model.

[0026] The intra-class and inter-class contrast alignment loss and the distribution alignment loss are respectively solved based on the local prototype with the same pseudo-label as the local augmented data, the first modality augmentation feature and the second modality augmentation feature corresponding to the local augmented data.

[0027] In some embodiments of this application, the intra-class and inter-class contrast alignment loss is solved based on indirect contrast loss and direct contrast loss;

[0028] The indirect contrast loss is solved based on the indirect contrast loss corresponding to the first mode and the second mode, which belong to different modal types respectively;

[0029] The indirect contrast loss corresponding to the first mode is solved based on the first mode enhancement feature, the local prototype with the same pseudo-label as the local enhancement data to which the first mode enhancement feature belongs, and a preset first temperature coefficient.

[0030] The indirect contrast loss corresponding to the second mode is solved based on the second mode enhancement feature, the local prototype with the same pseudo-label as the local enhancement data to which the second mode enhancement feature belongs, and the first temperature coefficient;

[0031] The direct comparison loss is solved based on a preset second temperature coefficient, the first modal enhancement feature and the second modal enhancement feature corresponding to the local enhancement data.

[0032] In some embodiments of this application, the distribution alignment loss is solved based on the loss of minimizing the difference between the pseudo-label distributions of the first modality enhancement feature and the second modality enhancement feature, and a preset weight hyperparameter.

[0033] The loss for minimizing the difference between the pseudo-label distributions of the first modality enhancement feature and the second modality enhancement feature is solved based on the pseudo-label distributions of the first modality enhancement feature, the second modality enhancement feature, and the average parameter; the value of the average parameter is the sum of the pseudo-label distributions of the first modality enhancement feature and the second modality enhancement feature.

[0034] The pseudo-label distribution of the first modality enhancement feature is solved based on the first modality enhancement feature and the local prototypes that have the same pseudo-labels as the local enhancement data to which the first modality enhancement feature belongs;

[0035] The pseudo-label distribution of the second modality enhancement feature is solved based on the second modality enhancement feature and the local prototype that has the same pseudo-label as the local enhancement data to which the second modality enhancement feature belongs.

[0036] A second aspect of this application provides a method for training a cross-modal retrieval model, comprising:

[0037] In the current federated learning round, local data is received from each client device. Each piece of local data contains: a corresponding local cross-modal retrieval model, a local prototype, and a local generator. The local cross-modal retrieval model, the local prototype, and the local generator are pre-obtained by the client device based on the cross-modal retrieval model training method provided in the first aspect above.

[0038] Global training data is obtained based on the local prototypes and local generators corresponding to the current federated learning round, and global cross-modal retrieval models corresponding to each local cross-modal retrieval model are trained based on the global training data.

[0039] In some embodiments of this application, the step of obtaining global training data based on each of the local prototypes and local generators corresponding to the current federated learning round, and training a global cross-modal retrieval model corresponding to each of the local cross-modal retrieval models based on the global training data, includes:

[0040] Cluster the local cross-modal retrieval models corresponding to the current federated learning round to obtain the corresponding global cross-modal retrieval model;

[0041] Cluster the local prototypes corresponding to the current federated learning round to obtain the corresponding global prototype;

[0042] Furthermore, based on the proportion of local multimodal data corresponding to each source, local augmentation data is generated by each of the global models to collectively constitute the corresponding global augmentation data;

[0043] Based on the global prototype, the global augmented data is filtered for discrete values ​​to obtain the corresponding global training data, and the global cross-modal retrieval model is trained using the global training data.

[0044] A third aspect of this application provides a training apparatus for a cross-modal retrieval model based on unsupervised federated learning, comprising:

[0045] An unsupervised training module is used to train a local adversarial generative network in an unsupervised manner, using the cross-modal splicing features corresponding to the local multimodal data and the local prototype obtained by clustering the cross-modal splicing features in advance as constraints, so as to train the adversarial generative network into a local generator for generating local augmented data.

[0046] The local data sending module is used to send the local prototypes, the local generators, and the local cross-modal retrieval models pre-trained locally based on the feature vectors of various modalities corresponding to the local multimodal data to the server, so that the server can obtain global training data based on the received local prototypes and local generators, and train the global cross-modal retrieval model corresponding to each local cross-modal retrieval model based on the global training data.

[0047] A fourth aspect of this application provides a training apparatus for a cross-modal retrieval model based on unsupervised federated learning, comprising:

[0048] The local data receiving module is used to receive local data from each client device in the current federated learning round. Each piece of local data contains: a local cross-modal retrieval model, a local prototype, and a local generator that correspond to each other. The local cross-modal retrieval model, the local prototype, and the local generator are obtained in advance by the client device based on the cross-modal retrieval model training method provided in the first aspect above.

[0049] The global model training module is used to obtain global training data based on the local prototypes and local generators corresponding to the current federated learning round, and to train the global cross-modal retrieval model corresponding to each local cross-modal retrieval model based on the global training data.

[0050] A fifth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-modal retrieval model training method provided in the first aspect, or implements the cross-modal retrieval model training method provided in the second aspect.

[0051] A sixth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal retrieval model training method provided in the first aspect above, or implements the cross-modal retrieval model training method provided in the second aspect above.

[0052] The seventh aspect of this application provides an unsupervised federated learning system, comprising: a server, and various client devices respectively communicatively connected to the server;

[0053] The client device includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the processor. When the first processor executes the first computer program, it implements the cross-modal retrieval model training method provided in the first aspect above.

[0054] The server includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the processor. When the second processor executes the second computer program, it implements the cross-modal retrieval model training method provided in the first aspect.

[0055] The eighth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the cross-modal retrieval model training method provided in the first aspect, or implements the cross-modal retrieval model training method provided in the second aspect.

[0056] The cross-modal retrieval model training method provided in this application uses cross-modal splicing features corresponding to local multimodal data and local prototypes obtained by pre-clustering the cross-modal splicing features as constraints to perform unsupervised training on a local adversarial generative network, thereby training the adversarial generative network into a local generator for generating local augmented data. The local prototypes, the local generators, and the local cross-modal retrieval models pre-trained locally based on feature vectors of various modalities corresponding to the local multimodal data are all sent to a server. The server obtains global training data based on the received local prototypes and local generators, and trains global cross-modal retrieval models corresponding to each local cross-modal retrieval model based on the global training data. This method enables unsupervised learning of local generators based on multimodal data during the training of global cross-modal retrieval models using network-based learning. It can fully utilize the multimodal relationships of local data without relying on high-quality data annotation, thereby improving the efficiency and reliability of global cross-modal retrieval model training while ensuring local data privacy, and effectively solving the offset problem of global cross-modal retrieval models.

[0057] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0058] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0059] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings:

[0060] Figure 1 This is a schematic diagram of the first process of a first cross-modal retrieval model training method in an embodiment of this application.

[0061] Figure 2 This is a schematic diagram of a second process of the first cross-modal retrieval model training method in one embodiment of this application.

[0062] Figure 3 This is a schematic diagram illustrating the execution flow of the training steps in the first cross-modal retrieval model training method according to an embodiment of this application.

[0063] Figure 4 This is a schematic diagram of the first process of the second cross-modal retrieval model training method in one embodiment of this application.

[0064] Figure 5 This is a schematic diagram of a second process for training a second cross-modal retrieval model in one embodiment of this application.

[0065] Figure 6 This is a schematic diagram of the structure of the first cross-modal retrieval model training device in one embodiment of this application.

[0066] Figure 7 This is a schematic diagram of the structure of a second cross-modal retrieval model training device in one embodiment of this application.

[0067] Figure 8 This is a schematic diagram illustrating the execution logic of the FedWGAN method for unsupervised cross-modal retrieval in an application example of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0069] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0070] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0071] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0072] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0073] To improve the efficiency and reliability of global cross-modal retrieval model training while ensuring local data privacy, embodiments of this application provide a first cross-modal retrieval model training method, a first cross-modal retrieval model training device for executing the first cross-modal retrieval model training method, a second cross-modal retrieval model training method, a second cross-modal retrieval model training device for executing the second cross-modal retrieval model training method, an electronic device, a computer-readable storage medium, an unsupervised federated learning system, and a computer program product. These methods, while ensuring local data privacy, fully utilize the multimodal relationships of local data during the global cross-modal retrieval model training process based on network learning, without relying on high-quality data annotation, thereby improving the efficiency and reliability of cross-modal retrieval model training.

[0074] The following examples will provide a detailed description.

[0075] Based on this, embodiments of this application provide a first cross-modal retrieval model training method that can be implemented by a first cross-modal retrieval model training device, see [link to relevant documentation]. Figure 1 The first cross-modal retrieval model training method specifically includes the following:

[0076] Step 100: Using the cross-modal splicing features corresponding to the local multimodal data and the local prototypes obtained by clustering the cross-modal splicing features in advance as constraints, perform unsupervised training on the local adversarial generative network to train the adversarial generative network into a local generator for generating local augmented data.

[0077] In one or more embodiments of this application, local multimodal data refers to the multimodal data local to the device where the first cross-modal retrieval model training device is located. This multimodal data may contain at least two different modal types of data, namely, first modal data and second modal data. One of the first modal data and the second modal data may be image data, and the other may be text data. Currently, cross-modality refers to the interaction, understanding, and processing of information between different modalities. A modality refers to the form of perception or data, such as text, images, audio, and video. Cross-modal research aims to achieve collaboration and interoperability between different modalities, such as cross-modal retrieval. Therefore, the first modal data and the second modal data may also contain audio and / or video data, selected specifically according to actual application requirements.

[0078] Taking image data and text data as examples, in one or more embodiments of this application, the first modality can refer to an image, and the second modality can refer to text; correspondingly, the first modality data can be image data, and the second modality data can be text data.

[0079] Correspondingly, the cross-modal splicing features corresponding to local multimodal data refer to the cross-modal splicing features obtained based on the local multimodal data. Specifically, a first modal feature vector can be extracted from the first modal data in the multimodal data, and a second modal feature vector can be extracted from the second modal data in the multimodal data. Then, cross-modal splicing features are generated based on the first modal feature vector and the second modal feature vector. In one example, the first modal feature vector can be an image feature vector (or simply image features); the second modal feature vector can be a text feature vector (or simply text features).

[0080] It is understood that the local prototype obtained by clustering the cross-modal splicing features refers to: performing clustering processing on each of the cross-modal splicing features using methods such as K-mans to obtain each cluster center, and then constructing a corresponding dataset from the clustering results corresponding to each cluster center as a local prototype of the first cross-modal retrieval model training device. From the perspective of the first cross-modal retrieval model training device, the local prototype can be called a local prototype.

[0081] In one or more embodiments of this application, the adversarial generative network may employ the WGAN (WassersteinGAN) model, which is an improved generative adversarial network (GAN) that addresses the problems of training stability, loss function indicativeness, and sample diversity in the original GAN.

[0082] It's also worth noting that federated learning is a privacy-preserving and distributed collaborative machine learning technique designed to train models without the data leaving the local machine. Its core idea is to distribute model training tasks across multiple devices or nodes, completing computation locally before sending model updates to a central server, without needing to centralize the original data. Data generation, on the other hand, is the process of synthesizing new data through algorithms or models, typically used to simulate real-world data or generate data for training and testing machine learning models.

[0083] Additionally, the local generator refers to the data generator located locally on the first cross-modal retrieval model training device, and can also be called a local generator or WGAN-generator. In the federated learning scenario, the first cross-modal retrieval model training device can be located on a client device, which communicates with the server in its federated learning system, and then sends the local data generated in step 100 in the current round of federated learning to the server through the following step 200.

[0084] Step 200: Send the local prototypes, the local generators, and the local cross-modal retrieval models trained locally based on the feature vectors of various modalities corresponding to the local multimodal data to the server, so that the server can obtain global training data based on the received local prototypes and local generators, and train the global cross-modal retrieval models corresponding to the local cross-modal retrieval models based on the global training data.

[0085] Understandably, from the perspective of the training device for the first cross-modal retrieval model, the local cross-modal retrieval model can also be called a local cross-modal retrieval model. Specifically, it can adopt the unsupervised contrastive cross-modal hashing (UCCH) model. The UCCH model implements learnable hashing operations through a momentum optimizer and proposes a cross-modal ranking learning loss (CRL) to utilize the discriminative power of all negative pairs, thereby improving retrieval performance. This method can perform cross-modal hashing learning better on unlabeled data, reducing the performance gap between contrastive learning and hashing.

[0086] In step 200, the server performs the following actions: receiving local data from different sources in the current federated learning round, wherein each local data contains: a corresponding local cross-modal retrieval model, a local prototype, and a local generator; clustering each local cross-modal retrieval model corresponding to the current federated learning round to obtain a corresponding global cross-modal retrieval model; clustering each local prototype corresponding to the current federated learning round to obtain a corresponding global prototype; and generating corresponding local augmentation data for each global model based on the proportion of local multimodal data from each source, which together constitute the corresponding global augmentation data; performing discrete value filtering on the global augmentation data based on the global prototype to obtain corresponding global training data, and using the global training data to train the global cross-modal retrieval model.

[0087] As can be seen from the above description, the first cross-modal retrieval model training method provided in this application embodiment can achieve unsupervised learning of local generators based on multimodal data during the global cross-modal retrieval model training process based on network learning. It can make full use of the multimodal relationships of local data without relying on high-quality data annotation. Thus, it can improve the efficiency and reliability of global cross-modal retrieval model training while ensuring the privacy of local data, and can effectively solve the offset problem of global cross-modal retrieval model.

[0088] To further improve the effectiveness and applicability of cross-modal retrieval model training, a first cross-modal retrieval model training method is provided in this application embodiment, see [link to relevant documentation]. Figure 2 The first cross-modal retrieval model training method includes the following content before step 100:

[0089] Step 010: Divide the local multimodal data into training data pairs, wherein each training data pair contains first modal data and second modal data belonging to different modal types.

[0090] Step 020: Input the first modality data in each training data pair into the feature extraction model corresponding to the first modality, and input the second modality data in each training data pair into the feature extraction model corresponding to the second modality, so that the feature extraction model corresponding to the first modality outputs the first modality feature vector corresponding to the first modality data, and the feature extraction model corresponding to the second modality outputs the second modality feature vector corresponding to the second modality data.

[0091] In one example, the first modal feature vector is an image feature vector F. IThe feature extraction model for the first modality can be a convolutional neural network (CNN) or a CNN-F model. CNN-F refers to a fast version of the convolutional neural network architecture, i.e., CNN-F (Fast), from the paper "Return of the Devil in the Details: Delving Deep into Convolutional Nets" (In BMVC, Chatfield, K.; Simonyan, K.; Vedaldi, A.; and Zisserman, A. 2014). The feature vector for the second modality is the text feature vector F. T The feature extraction model corresponding to the second modality can be the BERT (Bidirectional Encoder Representations from Transformers) model. The BERT model is a pre-trained language model based on the Transformer architecture. Through bidirectional encoder representation, it can simultaneously utilize the left and right contextual information of the input sequence, thereby more accurately understanding the meaning of the language.

[0092] Step 030: Train a local cross-modal retrieval model using the training data for the corresponding first modality feature vector, second modality data, and hash loss, so that the local cross-modal retrieval model can be used to obtain the hash representations corresponding to the first modality data and the second modality data respectively.

[0093] For the k-th client, its local cross-modal retrieval model can be written as UCCH. k .

[0094] And, step 040: Project and concatenate the first modality feature vector and the second modality data corresponding to each training data respectively, so as to obtain the cross-modal concatenated features corresponding to each training data respectively.

[0095] The projection of each training data point onto its corresponding first modality feature vector and second modality data can be performed using a multi-layer perceptron (MLP). In one example, the image feature vector F... I Input to the first multilayer perceptron (MLP) I The image feature vector F T Input to the second multilayer perceptron (MLP) T The cross-modal splicing feature is F, and the local prototype is C. For the k-th client, its local prototype can also be written as C. k .

[0096] Step 050: Cluster and project the corresponding cross-modal splicing features of each training data to obtain the local prototype corresponding to the local multimodal data.

[0097] To further improve the effectiveness and reliability of unsupervised training of local generators, a first cross-modal retrieval model training method is provided in this application embodiment, see [link to relevant documentation]. Figure 2 Step 100 in the first cross-modal retrieval model training method specifically includes the following:

[0098] Step 110: Use the cross-modal splicing features corresponding to the local multimodal data to perform multiple rounds of iterative training on the local adversarial generative network, and execute the preset training steps in each round.

[0099] Among them, see Figure 3 The training step in step 110 specifically includes the following:

[0100] Step 111: If the current iteration is the first round, input the cross-modal splicing features corresponding to the local multimodal data into the local adversarial generative network so that the adversarial generative network outputs the local augmented data for the current round; if the current iteration is not the first round, adjust the local adversarial generative network based on the training loss obtained in the previous round, and input the cross-modal splicing features corresponding to the local multimodal data into the adjusted adversarial generative network so that the adversarial generative network outputs the local augmented data for the current round.

[0101] The local augmentation data (or local augmentation data) can be denoted as F. g .

[0102] Step 112: Assign pseudo-labels to the local augmented data based on the local prototype.

[0103] And, step 113: divide the local augmentation data into first modal augmentation features and second modal augmentation features according to modal type.

[0104] In one example, the first modality enhancement feature is an image enhancement feature. The second modality enhancement feature is a text enhancement feature.

[0105] Step 114: Based on the first modality enhancement features and the second modality enhancement features corresponding to the local augmented data, and the local prototype with the same pseudo-label as the local augmented data, obtain the training loss with data quality constraints for the adversarial generative network in the next round.

[0106] In order to further realize unsupervised learning of local generators based on multimodal data and make full use of the multimodal relationships of local data without relying on high-quality data annotation, in the first cross-modal retrieval model training method provided in this application embodiment, the training loss with data quality constraints is solved based on the training loss function of the adversarial generative network.

[0107] The training loss function of the adversarial generative network includes: generator training loss function L. gen and discriminator training loss function L disc ;

[0108] The generator training loss function L gen Based on client loss L local And the preset generator loss L G Solve for the discriminator training loss function L. disc Based on the aforementioned client loss L local And the preset discriminator loss L D Solve this problem;

[0109] Wherein, the client loss L loca1 Based on intra-class and inter-class contrast alignment loss L c Distribution alignment loss L J And the hash loss L used to train the local cross-modal retrieval model h Solve this problem;

[0110] The intra-class and inter-class contrast alignment loss L c and the distribution alignment loss L J The solution is obtained based on the local prototype with the same pseudo-label as the local augmented data, the first modality augmentation feature and the second modality augmentation feature corresponding to the local augmented data, respectively.

[0111] To further improve the effectiveness and reliability of unsupervised learning of local generators, in a first cross-modal retrieval model training method provided in this application embodiment, the intra-class and inter-class contrast alignment loss L c Based on indirect comparison loss L ind Compared with direct comparison loss L d Solve this problem;

[0112] The indirect contrast loss L ind The solution is obtained by using the indirect contrast loss corresponding to the first and second modes, which belong to different modalities; for example, the indirect contrast loss based on image modality. Indirect contrast loss with text modality The solution is obtained.

[0113] The indirect contrast loss corresponding to the first modality is solved based on the first modality enhancement feature, the local prototype with the same pseudo-label as the local enhancement data to which the first modality enhancement feature belongs, and a preset first temperature coefficient; for example, based on the image modality enhancement feature. With the image modality enhancement features The local augmentation data F belongs to g The local prototype with the same pseudo-label and the preset first temperature coefficient σ1 are obtained by solving.

[0114] The indirect contrast loss corresponding to the second modality is solved based on the second modality enhancement feature, the local prototype with the same pseudo-label as the local enhancement data to which the second modality enhancement feature belongs, and the first temperature coefficient σ1; for example, based on the text modality enhancement feature. With the aforementioned text modality enhancement features The local augmentation data F belongs to g The local prototype with the same pseudo-label and the preset first temperature coefficient σ1 are obtained by solving.

[0115] The direct comparison loss L d The solution is obtained based on a preset second temperature coefficient σ2, the first modal enhancement features, and the second modal enhancement features corresponding to the local enhancement data; for example: based on the preset second temperature coefficient σ2, the local enhancement data F g The corresponding image modality enhancement features and the text modality enhancement features The direct comparison loss Ld is obtained by solving.

[0116] To further improve the effectiveness and reliability of unsupervised learning of local generators, in a first cross-modal retrieval model training method provided in this application embodiment, the distribution alignment loss L... J The loss is minimized based on the difference between the pseudo-label distributions of the first modality enhancement features and the second modality enhancement features, and the solution is obtained using preset weight hyperparameters; for example: based on image modality enhancement features and the text modality enhancement features The difference minimization loss is obtained by solving the preset weight hyperparameter α.

[0117] The loss for minimizing the difference between the pseudo-label distributions of the first modality enhancement feature and the second modality enhancement feature is calculated based on the pseudo-label distributions of the first modality enhancement feature, the second modality enhancement feature, and an average parameter. The average parameter is the sum of the pseudo-label distributions of the first and second modality enhancement features. For example, based on image modality enhancement features... Pseudo-label distribution The text modality enhancement feature Pseudo-label distribution And the loss minimization of difference is obtained by solving for the average parameter Z. Wherein, the average parameter Z is the pseudo-label distribution of the image modality enhancement features. and the pseudo-label distribution of the text modality enhancement features The sum of the average values.

[0118] The pseudo-label distribution of the first modality enhancement feature is solved based on the first modality enhancement feature and the local prototypes that have the same pseudo-labels as the local enhancement data to which the first modality enhancement feature belongs; for example, based on the image modality enhancement feature. and the image modality enhancement features The image modality enhancement features are obtained by solving the local prototypes of the local augmentation data that have the same pseudo-labels. Pseudo-label distribution

[0119] The pseudo-label distribution of the second modality enhancement feature is solved based on the second modality enhancement feature and the local prototypes that have the same pseudo-labels as the local enhancement data to which the second modality enhancement feature belongs, such as: based on the text modality enhancement feature. and the text modality enhancement features The local prototypes of the pseudo-labels belonging to the local augmented data are used to solve for the text modality augmentation features. Pseudo-label distribution

[0120] Furthermore, based on the aforementioned first cross-modal retrieval model training method, this application also provides a second cross-modal retrieval model training method that can be implemented by a second cross-modal retrieval model training device, which can be located in a server. (See [link to relevant documentation]). Figure 4 The second cross-modal retrieval model training method specifically includes the following:

[0121] Step 300: In the current federated learning round, local data is received from each client device, wherein each piece of local data contains: a local cross-modal retrieval model, a local prototype, and a local generator that correspond to each other; and the local cross-modal retrieval model, the local prototype, and the local generator are obtained in advance by the client device based on the first cross-modal retrieval model training method.

[0122] It is understood that the training method based on the first cross-modal retrieval model can be the first cross-modal retrieval model training method executed by the first cross-modal retrieval model training device set in the client device in the foregoing embodiments. Its specific implementation can refer to the embodiments of the foregoing first cross-modal retrieval model training method, which will not be repeated here.

[0123] Step 400: Obtain global training data based on each of the local prototypes and local generators corresponding to the current federated learning round, and train the global cross-modal retrieval model corresponding to each of the local cross-modal retrieval models based on the global training data.

[0124] As can be seen from the above description, the second cross-modal retrieval model training method provided in this application embodiment can achieve unsupervised learning of local generators based on multimodal data during the global cross-modal retrieval model training process based on network learning. It can make full use of the multimodal relationships of local data without relying on high-quality data annotation. Thus, it can improve the efficiency and reliability of global cross-modal retrieval model training while ensuring the privacy of local data, and can effectively solve the offset problem of global cross-modal retrieval model.

[0125] To further improve the effectiveness and reliability of global aggregation training and effectively address the offset problem of the global cross-modal retrieval model, the second cross-modal retrieval model training method provided in this application embodiment is as follows: Figure 5 Step 400 specifically includes the following:

[0126] Step 410: Cluster the local cross-modal retrieval models corresponding to the current federated learning round to obtain the corresponding global cross-modal retrieval model.

[0127] Step 420: Cluster the local prototypes corresponding to the current federated learning round to obtain the corresponding global prototype.

[0128] And, step 430: based on the proportion of local multimodal data corresponding to each source, generate corresponding local augmentation data through each global model to jointly constitute the corresponding global augmentation data.

[0129] Step 440: Based on the global prototype, perform discrete value filtering on the global augmented data to obtain the corresponding global training data, and use the global training data to train the global cross-modal retrieval model.

[0130] From a software perspective, this application also provides a first cross-modal retrieval model training apparatus for performing all or part of the first cross-modal retrieval model training method, see [link to relevant documentation]. Figure 6 The first cross-modal retrieval model training device specifically includes the following components:

[0131] The unsupervised training module 10 is used to train the local adversarial generative network in an unsupervised manner, using the cross-modal splicing features corresponding to the local multimodal data and the local prototype obtained by clustering the cross-modal splicing features in advance as constraints, so as to train the adversarial generative network into a local generator for generating local augmented data.

[0132] The local data sending module 20 is used to send the local prototype, the local generator, and the local cross-modal retrieval models pre-trained locally based on the feature vectors of various modalities corresponding to the local multimodal data to the server, so that the server can obtain global training data based on the received local prototypes and local generators, and train the global cross-modal retrieval model corresponding to each local cross-modal retrieval model based on the global training data.

[0133] The embodiments of the first cross-modal retrieval model training device provided in this application can be used to execute the processing flow of the embodiments of the first cross-modal retrieval model training method described above. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the first cross-modal retrieval model training method described above.

[0134] The cross-modal retrieval model training portion of the first cross-modal retrieval model training device can be completed in the client device. Specifically, the selection can be based on the processing capabilities of the client device and limitations of the user's usage scenario. This application does not impose any limitations in this regard. If all operations are completed in the client device, the client device may further include a processor for the specific processing of the cross-modal retrieval model training.

[0135] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0136] The server and the client device can communicate using any suitable network protocol, including those not yet developed as of the date of this application. Such network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Furthermore, such network protocols may also include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer Protocol) protocols used on top of the aforementioned protocols.

[0137] As can be seen from the above description, the first cross-modal retrieval model training device provided in this application embodiment can achieve unsupervised learning of the local generator based on multimodal data during the global cross-modal retrieval model training based on network learning. It can make full use of the multimodal relationships of local data without relying on high-quality data annotation. Thus, it can improve the efficiency and reliability of global cross-modal retrieval model training while ensuring the privacy of local data, and can effectively solve the offset problem of global cross-modal retrieval model.

[0138] Based on the global prototype, the global augmented data is filtered for discrete values ​​to obtain the corresponding global training data, and the global cross-modal retrieval model is trained using the global training data.

[0139] From a software perspective, this application also provides a second cross-modal retrieval model training apparatus for performing all or part of the second cross-modal retrieval model training method, see [link to relevant documentation]. Figure 7 The second cross-modal retrieval model training device specifically includes the following components:

[0140] The local data receiving module 30 is used to receive local data from each client device in the current federated learning round. Each piece of local data contains: a local cross-modal retrieval model, a local prototype, and a local generator that correspond to each other. The local cross-modal retrieval model, the local prototype, and the local generator are obtained in advance by the client device based on a cross-modal retrieval model training method that can be executed by the client.

[0141] The global model training module 40 is used to obtain global training data based on each of the local prototypes and local generators corresponding to the current federated learning round, and to train the global cross-modal retrieval model corresponding to each of the local cross-modal retrieval models based on the global training data.

[0142] The embodiments of the second cross-modal retrieval model training device provided in this application can be used to execute the processing flow of the embodiments of the second cross-modal retrieval model training method described above. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the second cross-modal retrieval model training method described above.

[0143] The training of the cross-modal retrieval model in the second cross-modal retrieval model training device can be completed on the server.

[0144] The server can communicate with client devices to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0145] As can be seen from the above description, the second cross-modal retrieval model training device provided in this application embodiment can achieve unsupervised learning of the local generator based on multimodal data during the global cross-modal retrieval model training based on network learning. It can make full use of the multimodal relationships of local data without relying on high-quality data annotation. Thus, it can improve the efficiency and reliability of global cross-modal retrieval model training while ensuring the privacy of local data, and can effectively solve the offset problem of global cross-modal retrieval model.

[0146] Furthermore, this application also provides an embodiment of an unsupervised federated learning system, comprising: a server, and various client devices that are respectively communicatively connected to the server;

[0147] The client device includes a first memory, a first processor, and a first computer program stored in the first memory and capable of running on the processor. When the first processor executes the first computer program, it implements the aforementioned first cross-modal retrieval model training method.

[0148] The server includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the processor. When the second processor executes the second computer program, it implements the aforementioned second modality retrieval model training method.

[0149] To further illustrate the above embodiments, this application also provides an application example of a cross-modal retrieval model training method implemented using an unsupervised federated learning system, namely a novel unsupervised cross-modal retrieval FedWGAN method. To alleviate the global model bias problem caused by non-independent and identically distributed data, this method fully utilizes the data from each client to obtain a global model with generalization and robustness. Utilizing client information to train the data generation model is a worthwhile approach, as each client can learn a data generator containing the characteristics of its own dataset to assist in the training of the global model. First, clustering methods are used to obtain a pseudo-label and a batch of local prototypes for the real data. Then, the class relationship and data distribution similarity between the real multimodal data and the generated multimodal data are used to further constrain the quality of the generated data to obtain a high-quality local generator. The locally trained generator is sent to the server to generate data. The local prototypes are also uploaded to the server to calculate the global cluster prototype, used to filter out outliers in the generated data on the server. Finally, the filtered data is input into the aggregated global retrieval model, using non-independent and identically distributed generated data to ensure that the bias problem of the final global model is resolved.

[0150] This application example illustrates the problem using image as the first modality and text as the second. Essentially, this example addresses cross-modal retrieval in a federated environment. Given a set of N image-text pairs... The multimodal dataset consists of I n and T n Let the nth sample be the image and the text, respectively. Cross-modal hash retrieval aims to obtain the hash representation of the image modality. and text modality Their respective hash representations, among which H represents the length of the final hash code. Vector similarity matching algorithms can be used for B and B. T Search between them.

[0151] The overall local training process and global training process are shown in Table 1 below:

[0152] Table 1

[0153]

[0154]

[0155] See Figure 8 On each client, images and text are first fed into CNN-F and BERT to extract their features F. I and F T Then, the concatenation result F is used to calculate the local prototype C, and provided to WGAN to generate the synthetic feature F. g Then synthesize feature F g Divided into and In order to improve and To improve WGAN's quality, it is further constrained by two loss functions proposed in this application example: intra-class and inter-class contrast loss for real and generated data, and JSD loss for data distribution alignment. After locally training WGAN and the retrieval model UCCH... k Then, this application example sends them along with the local prototype C to the server to compute the global prototype C. G And the global UCCH model. The uploaded WGAN generator is used to create synthetic data, and further filtered based on their minimum distance from the global prototype to remove outliers. The remaining synthetic data Used to train the global UCCH model on the server, the trained global model is sent back to the client to start the next round of communication.

[0156] This application example is mainly divided into four modules: a WGAN local trainer module, an intra-class and inter-class comparison alignment module, a local prototype-supervised distribution alignment module, and a global prototype-supervised global model correction module. This application example proposes an acronym, FedWGAN (Generating Synthetic Data for Unsupervised Federated Learning of Cross-Modal Retrieval), composed of Fed and WGAN. Fed represents federated learning, while WGAN represents the core WGAN network method used in this paper to implement an unsupervised cross-modal retrieval method guided by generated data in a federated scenario. The application example is described in detail module by module:

[0157] 1) WGAN Local Trainer Module

[0158] Since the client's private, non-IID data cannot be transmitted to the server, a data generator needs to be trained to simulate the original data, and then transmitted to the server. The generator should produce data with the same characteristics and distribution as the data on the client. Considering the stability and convergence of the training process, a Wasserstein Generative Adversarial Network (WGAN) is used to learn data generation. Similar to PT-FUCH, pre-trained CNNF and BERT are used to extract image features F. I (image features) and text features F T (Text features). Then, after MLP projection, they are concatenated into F (concatenated features):

[0159] F = concat(MLP) I (F I MLP T (F T )) (1)

[0160] WGAN consists of a generator (G) and a discriminator (D). The input to WGAN is Gaussian noise ∈ . The generator attempts to produce samples that are as realistic as possible from the noise to fool the discriminator. The traditional generator loss is defined as:

[0161]

[0162] This refers to noise ∈ in p ∈ Expectation under distribution

[0163] Where, p ∈ Let represent the distribution of noise ∈. The discriminator attempts to maximize the score of the true sample F while minimizing the score of the generated sample G(∈).

[0164]

[0165] p data This refers to the probability distribution of the actual input data; F~p data F refers to p data Obtained by sampling from the distribution;

[0166] The generator's output (i.e., the locally augmented data) is written as F g (i.e., generated feature), which should have a distribution similar to and identical to F. Local augmentation data F g The generated image enhancement features are divided into (i.e., generated image feature) and text enhancement features (i.e., generated text feature). In addition to the traditional WGAN loss function mentioned above, three new loss constraints were designed in the following two parts to further improve the quality of the generated data.

[0167] 2) Intra-class and inter-class comparison and alignment module

[0168] Since there are no data labels, K-means is first used to cluster the fused features F to obtain local class prototypes. ( This represents the feature representation of the Mth prototype in the kth client, where M is the number of prototypes and k represents the kth client. These prototypes are anchor points in the indirect constraint process, i.e., using these prototypes to generate local augmentation data F. g Assign pseudo-tags separately.

[0169] so and Pseudo-labels can then be used for indirect comparison constraints within and between classes. Taking the image modality as an example, image features assigned to the same pseudo-label should be close to each other and far from other prototypes. The same constraint applies to the text modality. This indirect loss can be expressed as:

[0170]

[0171] Represents image enhancement features The i-th sample (i.e., the i-th generated image feature);

[0172] Representative text enhancement features The i-th sample (i.e., the i-th generated text feature);

[0173] in Represents the sum on the k-th client Prototypes with the same pseudo-label, Sim represents the cosine similarity function, σ1 is the temperature coefficient, and i represents the feature of the i-th sample. These are summarized to obtain the indirect contrastive loss, which encourages features of different modalities to approach their corresponding local prototypes.

[0174]

[0175] Another point to note is that image and text features are generated simultaneously. Therefore, it's necessary to ensure they have the same label. A direct contrastive loss constraint is designed for this purpose:

[0176]

[0177] Represents image enhancement features The pseudo-label of the i-th sample;

[0178] Representative text enhancement features The pseudo-label of the i-th sample;

[0179] Here, σ² is also a temperature coefficient. The total loss of the intra-class and inter-class comparison alignment module is:

[0180] L c =L ind +L d (8)

[0181] 3) Distributed alignment module for local prototype supervision

[0182] To further align the generated image and text features, a loss based on JSD is designed from the perspective of their pseudo-label distribution. First, the pseudo-label distributions of the generated image and text features are calculated. and Where C is a local prototype on each client. The distribution of features is represented by the similarity between features and local prototypes C. Then, JSD divergence is used to measure the distribution. and The difference between the two distributions. Minimizing the difference between the two distributions enables the model to learn structural knowledge invariant to data augmentation. The formula for calculating this loss is:

[0183]

[0184] The same principle applies to the characteristics of real data, allowing for prototype improvement in subsequent iterations. The final result is the distribution alignment loss for local prototype supervision:

[0185]

[0186] (Q I ||Q T ) is a common notation used to represent the calculation of Q. I Q between T JSD divergence;

[0187] Here, α is a weight hyperparameter that gradually decreases from 1 to 0.5. This is because the generator's initial generation performance is not ideal, but it improves as training progresses. Therefore, the value of α is gradually increased as the generation performance improves.

[0188] According to the UCCH method, the hash loss L is used. hThe local unsupervised hash representation of the learning loss is used as the basis for the entire framework, and the summation yields the client loss L in this federated learning framework. local =L h +λL c +γL J , where λ and γ are hyperparameters used to control the weights of each loss. Let L local In the training process of WGAN, the generator training loss and the discriminator training loss can be respectively:

[0189] L gen =L local +L G (12)

[0190] L disc =L local +L D (13)

[0191] 4) Global prototype supervision global model correction module

[0192] This is because data generated by a local generator is used to assist the global model obtained after aggregation of local models. However, since the generated data is randomly produced by input noise, this synthetic data may not be consistent with the real data distribution. To improve the quality of the synthetic dataset, a global model correction module based on global prototype supervision is designed to improve the quality of the generated data and enhance the performance of the global model.

[0193] First, the generator G, local prototypes C, and local retrieval models from all clients are uploaded to the server. Then, the server aggregates the retrieval model UCCH and applies K-means to the uploaded local prototypes to obtain a set of global prototypes. On the other hand, the corresponding generator is generated on the server based on the number of samples on each client. Data points, where μ is a scaling hyperparameter that determines the number of samples generated on the server, D k This represents the number of samples on the k-th client.

[0194] Then, the synthetic data is filtered using the global prototype, retaining only data whose minimum distance to the global prototype is less than or equal to a threshold:

[0195]

[0196] This represents the global data generated on the server by the generator uploaded to the server. G here stands for global, g stands for generate, and i represents the i-th sample.

[0197] C represents the prototype of the m-th class globally, G represents the global representation, and m represents the clustering of samples from the m-th class.

[0198] Here, θ is a learnable parameter that determines and refines the threshold for filtering the synthetic data. This filtered data is used to train the global retrieval model. After training, the updated global retrieval model is sent back to each client for further training.

[0199] In other words, this application proposes a novel unsupervised cross-modal retrieval FedWGAN method, which alleviates the non-IID problem of data through a data generation strategy. It can benefit many existing federated learning methods. This application also designs two new loss constraints for WGAN to generate high-quality local client-side multimodal data in unsupervised scenarios. Furthermore, it designs a global outlier removal strategy on the server side to further improve the quality of the generated data.

[0200] In summary, the key point of this application example is that, in an unsupervised federated environment, by combining the intermodal relationships between real and generated features, a multimodal data generator is trained for each client that can generate data with features and distributions similar to those of the current client. Furthermore, after obtaining efficient data generators for each client, the client's local prototype, generator, and cross-modal retrieval model are all uploaded to the server. The local prototype is aggregated into a global prototype, and the local cross-modal retrieval models are aggregated into a global cross-modal retrieval model. The data generated by each client is then filtered using the global model, and the filtered data is used to retrain the global model to eliminate model bias.

[0201] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the first cross-modal retrieval model training method or the second cross-modal retrieval model training method mentioned in the above embodiments. The processor and the memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and the memory via wired or wireless means.

[0202] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0203] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the first or second cross-modal retrieval model training method in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the first or second cross-modal retrieval model training method in the above method embodiments.

[0204] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0205] The one or more modules are stored in the memory, and when executed by the processor, they execute the first cross-modal retrieval model training method or the second cross-modal retrieval model training method in the embodiment.

[0206] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0207] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0208] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0209] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the aforementioned first cross-modal retrieval model training method or second cross-modal retrieval model training method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0210] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned first cross-modal retrieval model training method or second cross-modal retrieval model training method.

[0211] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0212] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0213] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0214] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A cross-modal retrieval model training method, characterized in that, include: Using the cross-modal splicing features corresponding to local multimodal data and the local prototype obtained by clustering the cross-modal splicing features in advance as constraints, the local adversarial generative network is trained in unsupervised training so as to train the adversarial generative network into a local generator for generating local augmented data. The local prototypes, local generators, and local cross-modal retrieval models trained locally based on feature vectors of various modalities corresponding to local multimodal data are all sent to the server. This allows the server to obtain global training data based on the received local prototypes and local generators, and to train a global cross-modal retrieval model corresponding to each local cross-modal retrieval model based on the global training data.

2. The cross-modal retrieval model training method according to claim 1, characterized in that, Before performing unsupervised training on the local generative adversarial network, which is constrained by the cross-modal splicing features corresponding to the local multimodal data and the local prototype obtained by pre-clustering the cross-modal splicing features, the method further includes: The local multimodal data is divided into training data pairs, wherein each training data pair contains first modal data and second modal data belonging to different modal types respectively; The first modal data in each training data pair is input into the feature extraction model corresponding to the first modality, and the second modal data in each training data pair is input into the feature extraction model corresponding to the second modality, so that the feature extraction model corresponding to the first modality outputs the first modal feature vector corresponding to the first modal data, and the feature extraction model corresponding to the second modality outputs the second modal feature vector corresponding to the second modal data. The training data is used to train a local cross-modal retrieval model on the corresponding first modality feature vector, second modality data, and hash loss, so that the local cross-modal retrieval model can be used to obtain the hash representations corresponding to the first modality data and the second modality data respectively; Furthermore, each training data is projected onto and concatenated with its corresponding first modality feature vector and second modality data to obtain the cross-modal concatenated features corresponding to each training data. Clustering and projection are performed on the cross-modal splicing features corresponding to each of the training data to obtain the local prototype corresponding to the local multimodal data.

3. The cross-modal retrieval model training method according to claim 1, characterized in that, The step of unsupervised training of the local adversarial generative network (GCN) using cross-modal splicing features corresponding to local multimodal data and local prototypes obtained by pre-clustering the cross-modal splicing features as constraints, to train the GCN into a local generator for generating local augmented data, includes: The local adversarial generative network is iteratively trained in multiple rounds using cross-modal splicing features corresponding to local multimodal data, and preset training steps are executed in each round. The training steps include: If the current iteration is the first iteration, the cross-modal splicing features corresponding to the local multimodal data are input into the local adversarial generative network (GCN) so that the GCN outputs the local augmented data for the current iteration. If the current iteration is not the first iteration, the local GCN is adjusted based on the training loss obtained from the previous iteration, and the cross-modal splicing features corresponding to the local multimodal data are input into the adjusted GCN so that the GCN outputs the local augmented data for the current iteration. Assign pseudo-labels to the local augmented data based on the local prototype; Furthermore, the local augmentation data is divided into first-mode augmentation features and second-mode augmentation features according to modal type; Based on the first modality enhancement features and the second modality enhancement features corresponding to the local augmented data, and the local prototype with the same pseudo-label as the local augmented data, the training loss with data quality constraints corresponding to the adversarial generative network in the next round is obtained.

4. The cross-modal retrieval model training method according to claim 3, characterized in that, The training loss with data quality constraints is solved based on the training loss function of the adversarial generative network. The training loss function of the adversarial generative network includes: a generator training loss function and a discriminator training loss function; The generator training loss function is solved based on the client loss and a preset generator loss; the discriminator training loss function is solved based on the client loss and a preset discriminator loss. The client loss is calculated based on intra-class and inter-class contrast alignment loss, distribution alignment loss, and hash loss used to train the local cross-modal retrieval model. The intra-class and inter-class contrast alignment loss and the distribution alignment loss are respectively solved based on the local prototype with the same pseudo-label as the local augmented data, the first modality augmentation feature and the second modality augmentation feature corresponding to the local augmented data.

5. The cross-modal retrieval model training method according to claim 4, characterized in that, The intra-class and inter-class contrast alignment loss is solved based on indirect contrast loss and direct contrast loss; The indirect contrast loss is solved based on the indirect contrast loss corresponding to the first mode and the second mode, which belong to different modal types respectively; The indirect contrast loss corresponding to the first mode is solved based on the first mode enhancement feature, the local prototype with the same pseudo-label as the local enhancement data to which the first mode enhancement feature belongs, and a preset first temperature coefficient. The indirect contrast loss corresponding to the second mode is solved based on the second mode enhancement feature, the local prototype with the same pseudo-label as the local enhancement data to which the second mode enhancement feature belongs, and the first temperature coefficient; The direct comparison loss is solved based on a preset second temperature coefficient, the first modal enhancement feature and the second modal enhancement feature corresponding to the local enhancement data.

6. The cross-modal retrieval model training method according to claim 4, characterized in that, The distribution alignment loss is solved based on the loss of minimizing the difference between the pseudo-label distributions of the first modality enhancement feature and the second modality enhancement feature, as well as a preset weight hyperparameter. The loss for minimizing the difference between the pseudo-label distributions of the first modality enhancement feature and the second modality enhancement feature is solved based on the pseudo-label distributions of the first modality enhancement feature, the second modality enhancement feature, and the average parameter; the value of the average parameter is the sum of the pseudo-label distributions of the first modality enhancement feature and the second modality enhancement feature. The pseudo-label distribution of the first modality enhancement feature is solved based on the first modality enhancement feature and the local prototypes that have the same pseudo-labels as the local enhancement data to which the first modality enhancement feature belongs; The pseudo-label distribution of the second modality enhancement feature is solved based on the second modality enhancement feature and the local prototype that has the same pseudo-label as the local enhancement data to which the second modality enhancement feature belongs.

7. A cross-modal retrieval model training method, characterized in that, include: In the current federated learning round, local data is received from each client device, wherein each piece of local data contains: a corresponding local cross-modal retrieval model, a local prototype, and a local generator; and the local cross-modal retrieval model, the local prototype, and the local generator are pre-obtained by the client device based on the cross-modal retrieval model training method according to any one of claims 1 to 6. Global training data is obtained based on the local prototypes and local generators corresponding to the current federated learning round, and global cross-modal retrieval models corresponding to each local cross-modal retrieval model are trained based on the global training data.

8. The cross-modal retrieval model training method according to claim 7, characterized in that, The step of obtaining global training data based on each of the local prototypes and local generators corresponding to the current federated learning round, and training a global cross-modal retrieval model corresponding to each of the local cross-modal retrieval models based on the global training data, includes: Cluster the local cross-modal retrieval models corresponding to the current federated learning round to obtain the corresponding global cross-modal retrieval model; Cluster the local prototypes corresponding to the current federated learning round to obtain the corresponding global prototype; Furthermore, based on the proportion of local multimodal data corresponding to each source, local augmentation data is generated by each of the global models to collectively constitute the corresponding global augmentation data; Based on the global prototype, the global augmented data is filtered for discrete values ​​to obtain the corresponding global training data, and the global cross-modal retrieval model is trained using the global training data.

9. An unsupervised federated learning system, characterized in that, include: The server, and the various client devices that communicate with the server respectively; The client device includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the processor. When the first processor executes the first computer program, it implements the cross-modal retrieval model training method as described in any one of claims 1 to 6. The server includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the processor. When the second processor executes the second computer program, it implements the cross-modal retrieval model training method as described in claim 7 or 8.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the cross-modal retrieval model training method as described in any one of claims 1 to 6, or implements the cross-modal retrieval model training method as described in claim 7 or 8.

Images