A federated cross-modal retrieval method and system based on interactive prompts

Abstract

The application provides an interactive prompt-based federal cross-modal retrieval method and system, relates to the field of cross-modal information retrieval, uses first modal data to retrieve similar second modal data from a second modal data set based on the feature similarity of two modal data, and comprises the following steps: respectively extracting initial features of two modal data; using a cross-modal interaction network obtained through federated learning, taking a prompt vector as an intermediary, performing multi-layer bidirectional interaction between the initial features of two modal data, and obtaining final features of two modal data after cross-modal interaction; calculating the feature similarity based on the final features of two modal data, and screening similar second modal data; the application combines federated learning and prompt learning, improves the effectiveness and universality of cross-modal retrieval technology, and solves the problems of privacy protection and performance optimization in cross-modal retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cross-modal information retrieval, in particular to a federated cross-modal retrieval method and system based on interactive prompts. BACKGROUND

[0002] With the advancement of informatization and digitization, multi-modal data such as text, images and videos have surged, and cross-modal retrieval technology has shown important value in information retrieval, medical diagnosis and intelligent recommendation due to its ability to integrate multi-source heterogeneous data. Traditional cross-modal retrieval methods rely on centralized training, which faces high risk of data privacy leakage.

[0003] Federated learning is a distributed machine learning framework. Compared with traditional centralized machine learning, it performs model training on local devices and only transmits model updates rather than raw data, thereby realizing collaborative modeling while maintaining user privacy. The core mechanism of federated learning is to distribute computing tasks to edge devices where data resides, fully utilize local computing resources for model optimization, and integrate global model parameters through specific aggregation strategies. In mobile terminals, Internet of Things and medical data analysis, federated learning has shown significant practical value.

[0004] As a distributed training paradigm, federated learning effectively protects data privacy through local computation and parameter sharing, providing a new path for cross-modal retrieval. However, existing federated cross-modal retrieval research still faces challenges in retrieval performance, training efficiency and multi-modal data fusion, and innovative methods are needed to improve the utility and universality of the technology. SUMMARY

[0005] To solve the privacy protection and performance optimization problems in cross-modal retrieval, the present application proposes a federated cross-modal retrieval method and system based on interactive prompts, which combines federated learning and prompt learning to improve the utility and universality of cross-modal retrieval technology.

[0006] According to some embodiments, the present application adopts the following technical solutions:

[0007] A federated cross-modal retrieval method based on interactive prompts, which uses the similarity of features of two modal data to retrieve similar second modal data from the second modal data set using first modal data, comprising:

[0008] Respectively extracting initial features of two modal data;

[0009] Using the cross-modal interaction network obtained by federated learning, taking the prompt vector as an intermediary, performing multi-layer bidirectional interaction between the initial features of the two modal data, and obtaining the final features of the two modal data after cross-modal interaction;

[0010] The similarity calculation module is configured to calculate the feature similarity based on the final features of the two modalities, and filter the similar second modality data.

[0011] The cross-modality interaction network is composed of multiple prompt layers, each prompt layer has multiple prompt vectors, and the feature expression is refined in a hierarchical iterative manner through multi-level prompt learning, the deep semantic correlation between the features of the two modalities is gradually mined, and the cross-modality attention mechanism is calculated layer by layer to integrate the semantic information of the other modality into the feature representation of the current modality, and cross-modality interaction is realized.

[0012] According to some embodiments, the present application adopts the following technical scheme:

[0013] A federated cross-modality retrieval system based on interactive prompts, based on the feature similarity of two modal data, using the first modality data to retrieve similar second modality data from the second modality data set, comprising:

[0014] The feature extraction module is configured to extract initial features of two modal data respectively;

[0015] The feature interaction module is configured to use the cross-modality interaction network obtained through federated learning to perform multi-layer bidirectional interaction between the initial features of the two modalities using prompt vectors as intermediaries, and obtain the final features of the two modalities after cross-modality interaction;

[0016] The similarity calculation module is configured to calculate the feature similarity based on the final features of the two modalities, and filter the similar second modality data;

[0017] The cross-modality interaction network is composed of multiple prompt layers, each prompt layer has multiple prompt vectors, and the feature expression is refined in a hierarchical iterative manner through multi-level prompt learning, the deep semantic correlation between the features of the two modalities is gradually mined, and the cross-modality attention mechanism is calculated layer by layer to integrate the semantic information of the other modality into the feature representation of the current modality, and cross-modality interaction is realized.

[0018] According to some embodiments, the present application adopts the following technical scheme:

[0019] A computer program product comprising a computer program, the computer program being executed by a processor to implement the federated cross-modality retrieval method based on interactive prompts.

[0020] According to some embodiments, the present application adopts the following technical scheme:

[0021] A non-transitory computer readable storage medium for storing computer instructions, the computer instructions being executed by a processor to implement the federated cross-modality retrieval method based on interactive prompts.

[0022] According to some embodiments, the present application adopts the technical solutions as follows:

[0023] An electronic device comprises a processor, a memory and a computer program; wherein the processor is connected with the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the interactive prompt-based federal cross-modal retrieval method.

[0024] Compared with the prior art, the present application has the beneficial effects that:

[0025] (1) The present application constructs a multi-level cross-modal interaction network by integrating a learnable shared prompt vector and a federal learning framework, which can dynamically adjust image and text features and realize efficient training in a distributed environment.

[0026] (2) The present application introduces a cross-modal attention mechanism, which generates attention output by dot product attention, taking the features of the other modal as query and combining the prompt vector projected key and value, so as to realize the bidirectional interaction of image and text features in the encoding process.

[0027] (3) The present application proposes multi-level prompt learning to refine feature expression in a hierarchical iterative manner, gradually mining the deep semantic correlation between image and text features.

[0028] (4) The present application adopts orthogonal regularization mechanism to ensure the independence and diversity of the prompt vector, enhancing the robustness of the model to domain shift and data heterogeneity.

[0029] (5) The present application effectively reduces the risk of data privacy leakage by performing local calculation and global aggregation of the model under the federal learning framework. BRIEF DESCRIPTION OF DRAWINGS

[0030] The drawings accompanying the specification of the present application form a part thereof and serve to provide further understanding of the present application, the illustrative embodiments of the present application and its description serve to explain the present application, and do not constitute improper limitations on the present application.

[0031] Figure 1 Method flowchart for Example 1. DETAILED DESCRIPTION

[0032] The present application will be further described below in conjunction with the drawings and examples.

[0033] It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0034] It is to be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise, and it should be further understood that the terms "comprise" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, devices, components and / or combinations thereof.

[0035] Embodiment 1

[0036] In an embodiment of the present application, a federated cross-modal retrieval method based on interactive prompts is provided, which uses the first modal data to retrieve similar second modal data from the second modal data set based on the feature similarity of the two modal data, as shown in Figure 1 The method comprises the following steps:

[0037] Step S1: Extract the initial features of the two modal data respectively;

[0038] Step S2: Use the cross-modal interaction network obtained by federated learning to perform multi-layer bidirectional interaction between the initial features of the two modal data with the prompt vector as the intermediary, and obtain the final features of the two modal data after cross-modal interaction;

[0039] Step S3: Calculate the feature similarity based on the final features of the two modal data, and filter the similar second modal data;

[0040] The cross-modal interaction network is composed of multiple prompt layers, each prompt layer has multiple prompt vectors, and the feature expression is refined in a hierarchical iterative manner through multi-level prompt learning, gradually mining the deep semantic correlation between the features of the two modal data, and using a layer-by-layer cross-modal attention mechanism to calculate and integrate the semantic information of the opposite modal into the feature representation of the current modal, thereby realizing cross-modal interaction.

[0041] As an embodiment, the federated cross-modal retrieval method based on interactive prompts of the present application takes image-text retrieval as an example, and the initial features are extracted through a visual-linguistic pre-training model. The specific implementation process is described in detail as follows:

[0042] Visual-linguistic pre-training aims to learn the semantic association between images and texts by designing diverse pre-training objectives, and improve the performance of pre-trained models in visual and linguistic tasks. Contrastive learning is the core method of visual-linguistic pre-training, which learns the feature representation of images and texts by constructing discriminative objectives. Among them, CLIP (Contrastive Language-Image Pre-Training) uses large-scale image-text pairs for contrastive learning to map image and text features to a unified embedding space, and has achieved outstanding performance in cross-modal retrieval tasks, becoming one of the most representative visual-linguistic pre-training models.

[0043] Prompt learning is a lightweight fine-tuning method that guides pre-trained models to adapt to downstream tasks by designing specific input prompts, rather than directly adjusting model parameters. In recent years, it has shown significant application potential in the field of visual-linguistic pre-training, especially in cross-modal retrieval tasks. Prompt learning guides pre-trained models to perform retrieval tasks by designing specific prompts, which not only reduces the dependence on large-scale labeled datasets, but also significantly improves retrieval performance. Compared with the traditional fine-tuning paradigm, the unique advantage of prompt learning lies in its lightweight and modular characteristics, which can achieve task adaptation with only a small amount of prompt design or optimization, thereby greatly reducing the consumption of computing resources.

[0044] Although the above visual-linguistic pre-training and prompt learning have obvious advantages in improving training results and improving retrieval performance, this centralized mode has a high risk of data privacy leakage, which cannot be avoided.

[0045] In order to solve the deficiencies of existing cross-modal retrieval methods based on federated learning in retrieval performance and model training efficiency, and the high data privacy leakage risk of visual-linguistic pre-training models and prompt learning, this embodiment combines federated learning, visual-linguistic pre-training and prompt learning, and proposes a federated cross-modal retrieval method based on interactive prompts and visual-linguistic pre-training. The following two parts will be explained from the definition of cross-modal retrieval problem and the learning process of cross-modal retrieval model:

[0046] I. Definition of Cross-Modal Retrieval Problem

[0047] Generally, the task of cross-modal retrieval is to use data of one modality to retrieve data of another modality. In this embodiment, both image-to-text retrieval and text-to-image retrieval are supported, and the cross-modal retrieval model is constructed and optimized. The cross-modal retrieval model consists of three parts: an initial feature extraction model (i.e. visual-linguistic pre-training model CLIP), a cross-modal interaction network and a similarity calculation module, which correspond to steps S1, S2 and S3 above, respectively.

[0048] The multimodal dataset used for training is represented as follows: ,in, They represent the first Image data in a pair of image and text samples Text data, in each batch For the training samples, the initial image features extracted based on CLIP are: The initial text features are When cross-modal interaction networks contain When there are multiple cue layers, the set of cue vectors for each layer is represented as follows: （ ） The final image features and text features output by the cross-modal interaction network are represented as follows: and .

[0049] II. Cross-modal retrieval model learning process

[0050] The cross-modal retrieval model employs the Federated Avg algorithm as the core of the federated learning framework. It optimizes the global model through multiple rounds of communication. The pre-trained parameters of the visual-language pre-trained model CLIP remain frozen, so the optimization parameters of the federated learning only consist of the set of cue vectors for each cue layer. These are referred to as the optimized model parameters. .

[0051] In each round of communication, a subset of clients are randomly selected from all clients to participate in this round of training. The central server then sends the current global model parameters. ( (For the current communication round) is sent to the selected clients. Each client trains a model based on its local dataset and updates its local parameters. (k is the client number), the local model training process for each client is described below:

[0052] 1. Initial feature extraction model: Extract initial features from the two modalities respectively.

[0053] The initial image features and text features are generated by pre-trained CLIP, and are represented as follows: and .

[0054] 2. Cross-modal interaction network, using cue vectors as an intermediary for initial feature generation. and Multi-level bidirectional interaction between modalities yields the final features of the two modalities after cross-modal interaction. and .

[0055] The cross-modal interaction network has multiple prompt layers, each of which has multiple prompt vectors. In each prompt layer During the training process, a cross-modal attention mechanism is used. First, attention scores, i.e., weights of the prompt vectors, are calculated through bidirectional interaction of image and text features. Then, the attention output is obtained by weighted average of the multiple prompt vectors. Finally, the input features are dynamically adjusted using the attention output.

[0056] (1) Weight calculation and weighted average

[0057] When calculating the attention scores, a Dot-Product Attention form is used. Through the prompt vector as an intermediary, dynamic fusion between modalities is achieved, and the semantic information of the opposite modality is integrated into the current modality's feature representation.

[0058] Specifically, taking the image feature as an example, the text feature is taken as the query (Query), and the prompt vector generates the key (Key) and value (Value) after projection. The calculation process of the attention output is represented by the formula:

[0059]

[0060] where, is the text feature, is the prompt vector, represents the dot product of the query and the key, is a scaling factor to stabilize the gradient, is the normalization of the attention score , and the output is the weighted prompt vector representation, i.e., the attention output.

[0061] Similarly, the adjustment of the text feature takes the image feature as the query, and the calculation process of the attention output is represented by the formula:

[0062]

[0063] (2) Dynamic adjustment of input features using attention output

[0064] In multi-level prompt learning, the prompt vector of each layer further excavates specific semantic patterns based on the features of the previous layer, gradually refining the feature representation, thereby achieving adjustment of the features. Taking the image feature as an example, the image feature output by the first layer is:

[0065]

[0066] where, image features output by the previous layer, denote the text features for the query, by the first layer prompt vectors the computed attention outputs.

[0067] After the layer adjustment, the final image features are denoted as .

[0068] Similarly, the adjustment process of the text features is:

[0069]

[0070] The final text features are denoted as .

[0071] The loss function of the local model training process is composed of a contrastive loss, an orthogonal loss, and a norm loss; the contrastive loss aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs; the orthogonal loss aims to encourage the prompt vectors of each layer to be orthogonal to each other and to punish non-orthogonal vector pairs; the norm loss is used to constrain the length of each prompt vector to be close to the unit norm, which is described in detail below:

[0072] (1) Contrastive loss

[0073] The InfoNCE function is used to calculate the contrastive loss, aiming to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs.

[0074] Specifically, for image-to-text retrieval, the contrastive loss function can be written as:

[0075]

[0076] where is the batch size, is the temperature parameter, denotes the inner product of the features of the matching samples.

[0077] Similarly, the contrastive loss for text-to-image is represented as follows:

[0078]

[0079] The total contrastive loss is the average of the two, i.e. By optimizing this loss, the prompt vectors are adjusted to enhance the similarity of positive sample pairs, thereby improving the retrieval ability of the model.

[0080] (2) Orthogonal loss and norm loss

[0081] Multi-level prompt learning achieves deep alignment of image and text features through hierarchical iterative adjustment. However, if there is high similarity or redundancy between the prompt vectors of each layer, the expression ability of the feature adjustment process may be limited, and even the overfitting problem may be caused. Therefore, orthogonal regularization is introduced to constrain the distribution of the prompt vectors by defining an additional loss term.

[0082] For the prompt vector set of a certain layer The goal of orthogonal regularization is to make each prompt vector in as orthogonal as possible while maintaining unit norm. The loss function of the regularization loss can be divided into two parts: orthogonal loss and norm loss, which can be expressed by the formula:

[0083]

[0084] where represents the orthogonal loss between prompt vectors, is the norm loss.

[0085] is defined as:

[0086]

[0087] Here, and are the th and th prompt vectors in , the inner product of the two is , and the square term penalizes non-orthogonal vector pairs. Since the inner product between orthogonal vectors is zero, this term encourages vectors in to be orthogonal to each other, and only sums the upper triangular part to avoid repeated calculations.

[0088] The norm loss then constrains the length of each prompt vector to be close to the unit norm, which is defined as follows:

[0089]

[0090] where is the L2 norm of the th prompt vector, and the goal is to stabilize the norm of each vector around 1 to prevent the length of the vector from going out of control or tending to zero.

[0091] By jointly optimizing and , prompt orthogonal regularization ensures the independence and normalization of the prompt vectors.

[0092] (3) Total loss function

[0093] The above regularization loss is integrated into the total loss function of the model, which is optimized together with the contrastive loss, and the total loss is:

[0094]

[0095] where, is the contrastive loss in the form of InfoNCE, used to optimize the alignment of image-text features; is the weight hyperparameter of orthogonal regularization, used to balance the contribution of regularization term and main task loss.

[0096] During training, all levels of prompt vectors are subject to constraint, and the total regularization loss is the average of each layer loss, as follows:

[0097]

[0098] where, is the number of prompt layers, is the regularization loss of the layer prompt vector. By gradient descent optimization , the prompt vector gradually meets the orthogonality and norm constraints while maintaining the semantic alignment capability.

[0099] The optimization goal of the client is the total loss , and the trainable parameters are updated by the optimizer. At this point, the local model training process of the client is complete.

[0100] After each client completes the local model training, the updated parameters are uploaded to the central server, and the information transmitted is limited to the prompt vector. The pre-training parameters of CLIP remain frozen, thereby significantly reducing communication overhead.

[0101] After receiving all updates, the central server aggregates the new global parameters by weighted averaging:

[0102]

[0103] Subsequently, the clients participating in the next round of training are reselected, and the aggregated global parameters are distributed to each client, and the next round of training begins. The model uses a synchronous update strategy to ensure consistency of the global model after each round of training.

[0104] A specific federated learning example is provided, including the following steps:

[0105] Step 1: Randomly select a part of clients from all clients to participate in this round of training, and the central server distributes the current global model parameters to the selected clients.

[0106] Step 2: Each client trains the model based on its local multi-modal data set. The training process first extracts the initial image features and text features by the pre-trained CLIP feature extractor.

[0107] Step 3: Construct multiple prompt layers and initialize the prompt vectors of each layer. Starting from the first layer to the last layer, iteratively adjust the input image features and text features layer by layer. In each layer, calculate the attention score based on the prompt vector of the layer and the feature vector of the previous layer, and adjust the feature vector accordingly through the attention score to obtain the adjusted feature vector.

[0108] Step 4: After the iterative adjustment in step 3 is completed, calculate the contrastive loss using the adjusted final image features and text features.

[0109] Step 5: Apply regularization constraints to each layer's prompt vector, calculate the orthogonal loss and norm loss between each layer's prompt vectors respectively, and obtain the regularization loss of each layer. Weighted average the regularization loss of each layer to obtain the total regularization loss.

[0110] Step 6: Fuse the contrastive loss and the total regularization loss to obtain the total loss function, and optimize and update the local model parameters based on the total loss function.

[0111] Step 7: Each client uploads its updated parameters to the central server, and the server aggregates all client updates by weighted averaging to generate new global model parameters.

[0112] Step 8: Repeat the training process from step 1 to step 7 to gradually optimize the global model parameters. Input the task data that needs to be retrieved on the final model, and return the final retrieval result.

[0113] Embodiment 2

[0114] In an embodiment of the present application, a federated cross-modal retrieval system based on interactive prompts is provided, which uses the first modal data to retrieve similar second modal data from the second modal data set based on the feature similarity of the two modal data, comprising:

[0115] a feature extraction module configured to extract initial features of two modal data respectively;

[0116] The feature interaction module is configured to: utilize the cross-modal interaction network obtained through federated learning to perform multi-layer bidirectional interaction between the initial features of the two modalities with the prompt vector as an intermediary, and obtain the final features of the two modalities after cross-modal interaction.

[0117] The similarity calculation module is configured to: calculate feature similarity based on the final features of the two modalities, and screen similar second-modality data.

[0118] The cross-modal interaction network is composed of multiple prompt layers, each prompt layer has multiple prompt vectors, and the feature expression is refined in a hierarchical iteration manner through multi-level prompt learning, the deep semantic correlation between the features of the two modalities is gradually mined, and a layer-by-layer cross-modal attention mechanism is used for calculation, so that the semantic information of the opposite modality is integrated into the feature representation of the current modality, and cross-modal interaction is realized.

[0119] Embodiment 3

[0120] In an embodiment of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the federated cross-modal retrieval method based on interaction prompts.

[0121] Embodiment 4

[0122] In an embodiment of the present application, a non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the federated cross-modal retrieval method based on interaction prompts.

[0123] Embodiment 5

[0124] In an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the federated cross-modal retrieval method based on interaction prompts.

[0125] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions described in the flowcharts and / or block diagrams. Figure 1apparatus for performing the functions specified in the flow or flows and / or blocks. Figure 1 apparatus for performing the functions specified in the flow or flows and / or blocks.

[0126] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer implemented processing, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flow Figure 1 apparatus for performing the functions specified in the flow or flows and / or blocks. Figure 1 apparatus for performing the functions specified in the flow or flows and / or blocks.

[0127] The above describes the specific embodiments of the present application in conjunction with the drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. An interactive prompt-based federated cross-modal retrieval method, characterized in that, Retrieving similar second modality data from a second modality data set using the first modality data based on the feature similarity of the two modalities data, comprising: extracting initial features of the two modalities data respectively; using the cross-modal interaction network obtained by federated learning, taking the prompt vector as an intermediary, performing multi-layer bidirectional interaction between the initial features of the two modalities, and obtaining final features of the two modalities after cross-modal interaction; calculating the feature similarity based on the final features of the two modalities, and screening similar second modality data; wherein the cross-modal interaction network is composed of multiple prompt layers, each prompt layer has multiple prompt vectors, and the feature expression is refined in a hierarchical iterative manner through multi-level prompt learning, gradually mining the deep semantic correlation between the features of the two modalities, and adopting a layer-by-layer cross-modal attention mechanism to calculate and integrate the semantic information of the opposite modality into the feature representation of the current modality, thereby realizing cross-modal interaction; the prompt layer takes the two features output by the previous prompt layer as input, performs cross-modal interaction between the two features using the prompt vectors of the current prompt layer through cross-modal attention mechanism calculation, and outputs the two features after interaction, specifically: based on dot product attention, taking the feature of the opposite modality as the query, combining the projected key and value of the prompt vector, generating the weight of each prompt vector of the current prompt layer, and obtaining the attention output of the current modality through prompt vector weighted average, which can be expressed as: wherein, is a feature of the opposite modality, is a prompt vector, denotes the dot product of the query and the key, is a scaling factor, is a normalization of the attention scores, resulting in weights for the individual prompt vectors; Through multiple rounds of client local training and communication between the central server and the client, the loss function of the client local training is composed of contrast loss, orthogonal loss and norm loss.

2. The interactive prompt-based federated cross-modal retrieval method of claim 1, wherein, The two modalities are image and text, and the cross-modal retrieval includes two types of retrieval tasks: image-to-text retrieval, which takes image and text as first and second modalities respectively, and retrieves similar text from the text data set to the searched image; text-to-image retrieval, which takes text and image as first and second modalities respectively, and retrieves similar images from the image data set to the searched text.

3. The interactive prompt-based federated cross-modal retrieval method of claim 1, wherein, Add the current modality feature output by the previous prompt layer to the generated attention output to obtain the current modality feature after interaction of the current prompt layer.

4. The interactive prompt-based federated cross-modal retrieval method of claim 1, wherein, The federated learning of the cross-modal interaction network is based on a central server and several clients, and through multiple rounds of client local training and communication between the central server and the client, the prompt vectors of each prompt layer are globally optimized to obtain the trained cross-modal interaction network.

5. The interactive prompt-based federated cross-modal retrieval method of claim 4, wherein, The contrast loss aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs; The orthogonal loss aims to encourage the prompt vectors of each layer to be orthogonal to each other and to punish non-orthogonal vector pairs; The norm loss is used to constrain the length of each prompt vector to be close to the unit norm.

6. An interactive prompt-based federated cross-modal retrieval system employing an interactive prompt-based federated cross-modal retrieval method according to any one of claims 1-5. Retrieving similar second modality data from a second modality data set using the first modality data based on the feature similarity of the two modalities data, comprising: a feature extraction module configured to extract initial features of the two modalities data respectively; The feature interaction module is configured to use the cross-modal interaction network obtained through federated learning to perform multi-layer bidirectional interaction between the initial features of the two modalities using the prompt vector as an intermediary, and obtain the final features of the two modalities after cross-modal interaction. The similarity calculation module is configured to calculate feature similarity based on the final features of the two modalities, and screen similar second-modality data. The cross-modal interaction network is composed of multiple prompt layers, each prompt layer has multiple prompt vectors, and the feature expression is refined in a hierarchical iterative manner through multi-level prompt learning, the deep semantic correlation between the features of the two modalities is gradually mined, and a layer-by-layer cross-modal attention mechanism is used for calculation, the semantic information of the other modality is integrated into the feature representation of the current modality, and cross-modal interaction is realized.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the federated cross-modal retrieval method based on interactive prompts according to any one of claims 1-5.

8. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium is used to store computer instructions, and the computer instructions are executed by the processor to implement the federated cross-modal retrieval method based on interactive prompts according to any one of claims 1-5.

9. An electronic device, comprising: It comprises: A processor, a memory and a computer program; wherein the processor is connected with the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the federated cross-modal retrieval method based on interactive prompts according to any one of claims 1-5.

Images