Federal cross-modal retrieval method and system based on interaction prompt

Abstract

The invention provides a federal cross-modal retrieval method and system based on interactive prompts, and relates to the field of cross-modal information retrieve.The federal cross-modal retrieval method and system based on interactive prompts retrieve similar second modal data from a second modal data set by using first modal data based on the feature similarity of two modal data comprises the steps that initial features of the two modal data are extracted respectively; performing multi-layer bidirectional interaction between the initial features of the two modals by using a cross-modal interaction network obtained by federal learning and taking a prompt vector as an intermediary to obtain final features of the two modals after cross-modal interaction; calculating feature similarity based on the final features of the two modalities, and screening similar second modal data; according to the method, federal learning and prompt learning are combined, so that the effectiveness and universality of a cross-modal retrieval technology are improved, and the problems of privacy protection and performance optimization in cross-modal retrieval are solved.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal information retrieval, and in particular to a federated cross-modal retrieval method and system based on interactive prompting. Background Art

[0002] With the advancement of informatization and digitalization, multimodal data such as text, images, and videos has surged. Cross-modal retrieval technology, capable of integrating heterogeneous data from multiple sources, has demonstrated significant value in fields such as information retrieval, medical diagnosis, and intelligent recommendations. Traditional cross-modal retrieval methods rely heavily on centralized training, which carries a high risk of data privacy leaks.

[0003] Federated learning is a distributed machine learning framework. Compared to traditional centralized machine learning, it performs model training locally on devices, transmitting only model updates rather than raw data. This enables collaborative modeling while maintaining user privacy. The core mechanism of federated learning is to distribute computing tasks to edge devices where data resides, fully utilizing the local computing resources of each device for model optimization and integrating global model parameters through specific aggregation strategies. Federated learning has demonstrated significant practical value in application areas such as mobile devices, the Internet of Things, and medical data analysis.

[0004] As a distributed training paradigm, federated learning effectively protects data privacy through local computing and parameter sharing, providing a new path for cross-modal retrieval. However, existing research on federated cross-modal retrieval still faces challenges in terms of retrieval performance, training efficiency, and multimodal data fusion. Innovative methods are urgently needed to improve the effectiveness and universality of the technology. Summary of the Invention

[0005] In order to solve the problems of privacy protection and performance optimization in cross-modal retrieval, the present invention proposes a federated cross-modal retrieval method and system based on interactive prompts, which combines federated learning and prompt learning to improve the effectiveness and universality of cross-modal retrieval technology.

[0006] According to some embodiments, the present invention adopts the following technical solutions: A federated cross-modal retrieval method based on interactive prompts uses first modality data to retrieve similar second modality data from a second modality dataset based on feature similarity between the two modal data, including: Extract the initial features of the two modal data respectively; Using the cross-modal interaction network obtained by federated learning and the prompt vector as an intermediary, a multi-layer bidirectional interaction is performed between the initial features of the two modalities to obtain the final features of the two modalities after the cross-modal interaction; Based on the final features of the two modalities, the feature similarity is calculated and similar second modality data is screened; Among them, the cross-modal interaction network consists of multiple prompt layers, each prompt layer has multiple prompt vectors. Through multi-level prompt learning, the feature expression is refined in a hierarchical iterative manner, and the deep semantic association between the two modal features is gradually explored. The layer-by-layer cross-modal attention mechanism is used for calculation to integrate the semantic information of the other modality into the feature representation of the current modality to achieve cross-modal interaction.

[0007] According to some embodiments, the present invention adopts the following technical solutions: A federated cross-modal retrieval system based on interactive prompts uses first modality data to retrieve similar second modality data from a second modality dataset based on feature similarity between the two modal data, including: The feature extraction module is configured to: extract initial features of the two modal data respectively; The feature interaction module is configured to: utilize the cross-modal interaction network obtained through federated learning, using the prompt vector as an intermediary, to perform multi-layer bidirectional interaction between the initial features of the two modalities, and obtain the final features of the two modalities after the cross-modal interaction; The similarity calculation module is configured to: calculate feature similarity based on the final features of the two modalities and screen similar second modal data; Among them, the cross-modal interaction network consists of multiple prompt layers, each prompt layer has multiple prompt vectors. Through multi-level prompt learning, the feature expression is refined in a hierarchical iterative manner, and the deep semantic association between the two modal features is gradually explored. The layer-by-layer cross-modal attention mechanism is used for calculation to integrate the semantic information of the other modality into the feature representation of the current modality to achieve cross-modal interaction.

[0008] According to some embodiments, the present invention adopts the following technical solutions: A computer program product includes a computer program, which implements the interactive prompt-based federated cross-modal retrieval method when executed by a processor.

[0009] According to some embodiments, the present invention adopts the following technical solutions: A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method of federated cross-modal retrieval based on interactive prompts is implemented.

[0010] According to some embodiments, the present invention adopts the following technical solutions: An electronic device includes: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the described federated cross-modal retrieval method based on interactive prompts.

[0011] Compared with the prior art, the present invention has the following beneficial effects: (1) By integrating a learnable shared cue vector and a federated learning framework, this paper constructs a multi-level cross-modal interaction network that can dynamically adjust image and text features and achieve efficient training in a distributed environment.

[0012] (2) The present invention introduces a cross-modal attention mechanism, which uses dot product attention to take the features of the other modality as the query and combines the key and value of the projected prompt vector to generate the attention output, thereby realizing the two-way interaction between image and text features in the encoding process.

[0013] (3) This paper proposes multi-level prompt learning to refine feature expression in a hierarchical and iterative manner, and gradually explore the deep semantic associations between image and text features.

[0014] (4) The present invention adopts an orthogonal regularization mechanism to ensure the independence and diversity of the cue vectors, thereby enhancing the robustness of the model to domain shift and data heterogeneity.

[0015] (5) This invention effectively reduces the risk of data privacy leakage by performing local calculation and global aggregation of the model under the federated learning framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0017] Figure 1 This is a flow chart of the method of Example 1. DETAILED DESCRIPTION The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0019] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "comprising" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0020] Example 1 In one embodiment of the present invention, a federated cross-modal retrieval method based on interactive prompts is provided, which uses first modality data to retrieve similar second modality data from a second modality dataset based on feature similarity of the two modal data. Figure 1 As shown, including: Step S1: extracting initial features of the two modal data respectively; Step S2: Using the cross-modal interaction network obtained by federated learning, with the prompt vector as the intermediary, a multi-layer bidirectional interaction is performed between the initial features of the two modalities to obtain the final features of the two modalities after the cross-modal interaction; Step S3: Based on the final features of the two modalities, calculate the feature similarity and screen similar second modal data; Among them, the cross-modal interaction network consists of multiple prompt layers, each prompt layer has multiple prompt vectors. Through multi-level prompt learning, the feature expression is refined in a hierarchical iterative manner, and the deep semantic association between the two modal features is gradually explored. The layer-by-layer cross-modal attention mechanism is used for calculation to integrate the semantic information of the other modality into the feature representation of the current modality to achieve cross-modal interaction.

[0021] As an embodiment, the present invention provides a federated cross-modal retrieval method based on interactive prompts. Taking image-text retrieval as an example, initial features are extracted through a vision-language pre-training model. The specific implementation process is described in detail below: Vision-language pre-training aims to improve the performance of pre-trained models in vision and language tasks by designing diverse pre-training objectives to learn the semantic associations between images and text. Contrastive learning is a core method in vision-language pre-training, which learns feature representations of images and text by constructing discriminative objectives. Among them, CLIP (Contrastive Language-Image Pre-Training) leverages contrastive learning of large-scale image-text pairs to map image and text features into a unified embedding space. It has demonstrated outstanding performance in tasks such as cross-modal retrieval, becoming one of the most representative vision-language pre-training models.

[0022] Prompt learning is a lightweight fine-tuning method that guides pre-trained models to adapt to downstream tasks by designing specific input prompts, rather than directly adjusting model parameters. In recent years, it has shown significant application potential in vision-language pre-training, particularly in cross-modal retrieval tasks. By designing specific prompts to guide pre-trained models in retrieval tasks, prompt learning not only reduces the reliance on large-scale annotated datasets but also significantly improves retrieval performance. Compared to traditional fine-tuning paradigms, prompt learning offers a unique advantage in its lightweight and modular nature. Task adaptation requires only minimal prompt design or optimization, significantly reducing computational resource consumption.

[0023] Although the above-mentioned visual-language pre-training and prompt learning have obvious advantages in improving training effects and retrieval efficiency, this centralized model has a high risk of data privacy leakage, which is an unavoidable problem.

[0024] To address the shortcomings of existing cross-modal retrieval methods based on federated learning in terms of retrieval performance and model training efficiency, as well as the high data privacy leakage risks associated with vision-language pre-training models and prompt learning, this embodiment combines federated learning, vision-language pre-training, and prompt learning to propose a federated cross-modal retrieval method based on interactive prompts and vision-language pre-training. The following describes the cross-modal retrieval problem definition and the cross-modal retrieval model learning process. 1. Definition of Cross-modal Retrieval Problem Generally, the task of cross-modal retrieval is to use data from one modality to retrieve data from another modality. In this embodiment, two types of retrieval tasks, image-to-text retrieval and text-to-image retrieval, are supported to construct and optimize a cross-modal retrieval model. The cross-modal retrieval model here consists of three parts: an initial feature extraction model (i.e., the visual-language pre-training model CLIP), a cross-modal interaction network, and a similarity calculation module, whose functions correspond to steps S1, S2, and S3 above, respectively.

[0025] The multimodal dataset used for training is represented as ,in, Respectively represent Image data in image-text pairs Text data, in each batch For the training samples, the initial image features extracted based on CLIP are , the initial text features are When the cross-modal interaction network contains When there are multiple prompt layers, the set of prompt vectors for each layer is expressed as （ ）, the final image features and text features output by the cross-modal interaction network are expressed as and .

[0026] 2. Cross-modal retrieval model learning process The cross-modal retrieval model uses the Federated Averaging Algorithm (FedAvg) as the core of the federated learning framework. It optimizes the global model through multiple rounds of communication. The pre-trained parameters of the visual-language pre-training model CLIP remain frozen, so the optimization parameters of federated learning are only the set of prompt vectors of each prompt layer. , which are called optimized model parameters .

[0027] In each round of communication, some clients are randomly selected from all clients as participants in this round of training, and the central server sends the current global model parameters ( is the current communication round) is sent to the selected clients, each client trains the model based on the local dataset and updates the local parameters to (k is the client serial number). The following describes the local model training process of each client: 1. Initial feature extraction model, extracting the initial features of the two modal data respectively.

[0028] The initial image features and text features are generated by pre-trained CLIP, respectively, and are expressed as and .

[0029] 2. Cross-modal interaction network, using prompt vector as intermediary to perform initial feature and The final features of the two modalities after cross-modal interaction are obtained through multi-layer bidirectional interaction between and .

[0030] The cross-modal interaction network has multiple prompt layers, and each layer has multiple prompt vectors. During the training process, a cross-modal attention mechanism is adopted. First, the attention score, that is, the weight of each prompt vector, is calculated through the bidirectional interaction of image and text features. Then, multiple prompt vectors are weighted averaged to obtain the attention output. Finally, the attention output is used to dynamically adjust the input features.

[0031] (1) Weight calculation and weighted average When calculating the attention score, the dot-product attention form is used, and the prompt vector is used as an intermediary to achieve dynamic fusion between modalities and integrate the semantic information of the other modality into the feature representation of the current modality.

[0032] Specifically, taking image features as an example, text features As a query, the hint vector After projection, the key and value are generated respectively. The calculation process of attention output is expressed as follows:

[0033] in, is the text feature, is the prompt vector, represents the dot product of query and key, is the scaling factor to stabilize the gradient, is the attention score The normalization of , the output is the weighted prompt vector representation, that is, the attention output. Similarly, the text features are adjusted based on the image features. As a query, the calculation process of the attention output is expressed as:

[0034] (2) Use attention output to dynamically adjust input features In multi-level cue learning, the cue vector of each layer Based on the features of the previous layer, we further explore specific semantic patterns and gradually refine the feature representation to achieve feature adjustment. Taking image features as an example, the first The image features output by the layer are:

[0035] in, is the image feature output by the previous layer, Represents text features To query, through Layer hint vector Computed attention output.

[0036] go through After layer adjustment, the final image features are , recorded as .

[0037] Similarly, the text feature adjustment process is:

[0038] The final text features are , recorded as .

[0039] The loss function of the local model training process consists of contrast loss, orthogonal loss, and norm loss. The contrast loss aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The orthogonal loss encourages the pairwise orthogonality of the hint vectors of each layer and penalizes non-orthogonal vector pairs. The norm loss is used to constrain the length of each hint vector to be close to the unit norm. The following details are explained in detail: (1) Contrastive loss The InfoNCE function is used to calculate the contrast loss, with the goal of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs.

[0040] Specifically, for image-to-text retrieval, the contrastive loss function can be written as:

[0041] in, is the batch size, is the temperature parameter, Indicates the The inner product of the features of the matching samples.

[0042] Similarly, the contrastive loss for text to image is expressed as follows:

[0043] The total contrast loss is the average of the two, i.e. By optimizing this loss, the entire hint vector is adjusted to enhance the similarity of positive sample pairs, thereby improving the retrieval ability of the model.

[0044] (2) Orthogonal loss and norm loss Multi-level cue learning achieves deep alignment of image and text features through hierarchical iterative adjustment. However, if there is high similarity or redundancy between the cue vectors of each layer, the expressive power of the feature adjustment process may be limited, and even cause overfitting problems. Therefore, orthogonal regularization is introduced to impose constraints on the distribution of cue vectors by defining additional loss terms.

[0045] For a certain layer of prompt vector set , the goal of orthogonal regularization is to make The prompt vectors in are as orthogonal as possible while maintaining the unit norm. The loss function of the regularization loss can be divided into two parts: orthogonal loss and norm loss, which can be expressed as follows:

[0046] in, represents the orthogonal loss between the prompt vectors, is the norm loss.

[0047] Defined as:

[0048] here, and for The and Hint vectors, is the inner product of the two, the square term Penalize non-orthogonal vector pairs. Since the inner product between orthogonal vectors is zero, this encourages The vectors in are orthogonal to each other, only for the upper triangular part Sum to avoid double counting.

[0049] Norm loss Then constrain the length of each hint vector to be close to the unit norm, defined as follows:

[0050] in, For the The goal is to stabilize the norm of each vector near 1 to prevent the vector length from getting out of control or approaching zero.

[0051] Through joint optimization and ,Cue orthogonality regularization ensures the independence and normalization of the cue vectors.

[0052] (3) Total loss function The above regularization loss is integrated into the total loss function of the model and optimized together with the contrast loss. The total loss is:

[0053] in, InfoNCE is a contrast loss in the form of InfoNCE, which is used to optimize the alignment of image-text features; is the weight hyperparameter of the orthogonal regularization, which is used to balance the contribution of the regularization term and the main task loss.

[0054] During training, all layers of the prompt vectors are applied Constraint, its total regularization loss is the average of the losses of each layer, as shown below:

[0055] in, To indicate the number of layers, For the Regularization loss of the layer hint vector. Optimized by gradient descent ,The hint vector gradually satisfies the orthogonality and norm constraints while maintaining the semantic alignment capability.

[0056] The client's optimization goal is total loss , the trainable parameters are updated through the optimizer. This concludes the introduction to the client's local model training process.

[0057] After each client completes local model training, it uploads the updated parameters to the central server. The information transmitted is limited to the prompt vector, and the pre-trained parameters of CLIP remain frozen, which significantly reduces communication overhead.

[0058] After receiving all updates, the central server generates new global parameters through weighted average aggregation:

[0059] Then, reselect the clients to participate in the next round of training and use the aggregated global parameters After being sent to each client and entering the next round of training, the model adopts a synchronous update strategy to ensure that the global model remains consistent after each round of training.

[0060] Provide a specific federated learning example, including the following steps: Step 1: Randomly select some clients from all clients to participate in this round of training, and the central server sends the current global model parameters to the selected clients.

[0061] Step 2: Each client performs model training based on its local multimodal dataset. The training process first uses the pre-trained CLIP feature extractor to extract image features and text features respectively to obtain initial image features and initial text features.

[0062] Step 3: Construct multiple hint layers and initialize the hint vectors for each layer. Starting from the first layer to the last layer, the input image and text features are adjusted layer by layer. At each layer, the attention score is calculated based on the hint vector of that layer and the feature vector of the previous layer. The feature vector is adjusted accordingly based on the attention score to obtain the adjusted feature vector.

[0063] Step 4: After the layer-by-layer iterative adjustment in step 3 is completed, the contrast loss is calculated using the adjusted final image features and text features.

[0064] Step 5: Apply regularization constraints to the hint vectors of each layer, calculate the orthogonal loss and norm loss between the hint vectors of each layer respectively, obtain the regularization loss of each layer, and perform weighted average of the regularization losses of each layer to obtain the total regularization loss.

[0065] Step 6: The contrast loss and the total regularization loss are combined to obtain the total loss function. The client optimizes and updates the local model parameters based on the total loss function.

[0066] Step 7: Each client uploads its updated parameters to the central server, which aggregates the updates of all clients through weighted averaging to generate new global model parameters.

[0067] Step 8: Repeat the training process from Steps 1 to 7 to gradually optimize the global model parameters. Input the task data to be retrieved into the final model and return the final retrieval results.

[0068] Example 2 In one embodiment of the present invention, a federated cross-modal retrieval system based on interactive prompts is provided. Based on feature similarity between the two modal data, first modal data is used to retrieve similar second modal data from a second modal dataset, including: The feature extraction module is configured to: extract initial features of the two modal data respectively; The feature interaction module is configured to: utilize the cross-modal interaction network obtained through federated learning, using the prompt vector as an intermediary, to perform multi-layer bidirectional interaction between the initial features of the two modalities, and obtain the final features of the two modalities after the cross-modal interaction; The similarity calculation module is configured to: calculate feature similarity based on the final features of the two modalities and screen similar second modal data; Among them, the cross-modal interaction network consists of multiple prompt layers, each prompt layer has multiple prompt vectors. Through multi-level prompt learning, the feature expression is refined in a hierarchical iterative manner, and the deep semantic association between the two modal features is gradually explored. The layer-by-layer cross-modal attention mechanism is used for calculation to integrate the semantic information of the other modality into the feature representation of the current modality to achieve cross-modal interaction.

[0069] Example 3 In one embodiment of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the aforementioned federated cross-modal retrieval method based on interactive prompts.

[0070] Example 4 In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which is used to store computer instructions. When the computer instructions are executed by a processor, the interactive prompt-based federated cross-modal retrieval method is implemented.

[0071] Example 5 In one embodiment of the present invention, an electronic device is provided, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the described federated cross-modal retrieval method based on interactive prompts.

[0072] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0074] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A federated cross-modal retrieval method based on interactive prompts, characterized in that: Based on feature similarity of the two modal data, similar second modal data is retrieved from a second modal dataset using the first modal data, comprising: Extract the initial features of the two modal data respectively; Using the cross-modal interaction network obtained by federated learning and the prompt vector as an intermediary, a multi-layer bidirectional interaction is performed between the initial features of the two modalities to obtain the final features of the two modalities after the cross-modal interaction; Based on the final features of the two modalities, the feature similarity is calculated and similar second modality data is screened; Among them, the cross-modal interaction network consists of multiple prompt layers, each prompt layer has multiple prompt vectors. Through multi-level prompt learning, the feature expression is refined in a hierarchical iterative manner, and the deep semantic association between the two modal features is gradually explored. The layer-by-layer cross-modal attention mechanism is used for calculation to integrate the semantic information of the other modality into the feature representation of the current modality to achieve cross-modal interaction.

2. The method for federated cross-modal retrieval based on interactive prompting according to claim 1, characterized in that: The two modalities are image and text, and the cross-modal retrieval includes two types of retrieval tasks: Image-to-text retrieval uses images and text as the first and second modalities, respectively, to retrieve texts similar to the image being retrieved from a text dataset. Text-to-image retrieval uses text and image as the first and second modalities respectively, and retrieves images similar to the retrieved text from the image dataset.

3. The method for federated cross-modal retrieval based on interactive prompting according to claim 1, characterized in that: The prompt layer takes the two features output by the previous prompt layer as input, calculates through the cross-modal attention mechanism, uses the prompt vector of the current prompt layer, performs cross-modal interaction between the two features, and outputs the two features after the interaction, specifically: Based on dot product attention, the features of the other modality are used as queries, and the weights of each hint vector of the current hint layer are generated by combining the keys and values ​​of the projected hint vectors. The attention output of the current modality is obtained by weighted averaging the hint vectors. The current modality features output by the previous prompt layer are added to the generated attention output to obtain the current modality features after interaction of the current prompt layer.

4. The method for federated cross-modal retrieval based on interactive prompting according to claim 3, characterized in that: The attention output of the current modality is expressed as follows: in, is the characteristic of the other mode, is the prompt vector, represents the dot product of query and key, is the scaling factor, It is to normalize the attention scores and obtain the weights of each prompt vector.

5. The method for federated cross-modal retrieval based on interactive prompting according to claim 1, characterized in that: The federated learning of the cross-modal interaction network is based on a central server and several clients. Through multiple rounds of client local training and communication between the central server and the clients, the prompt vector of each prompt layer is globally optimized to obtain a trained cross-modal interaction network.

6. The method for federated cross-modal retrieval based on interactive prompting according to claim 5, characterized in that: The loss function of the client local training consists of contrast loss, orthogonal loss and norm loss; The contrast loss aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs; The orthogonal loss aims to encourage the prompt vectors of each layer to be pairwise orthogonal and penalize non-orthogonal vector pairs; The norm loss is used to constrain the length of each hint vector to be close to the unit norm.

7. A federated cross-modal retrieval system based on interactive prompts, characterized by: Based on feature similarity of the two modal data, similar second modal data is retrieved from a second modal dataset using the first modal data, comprising: The feature extraction module is configured to: extract initial features of the two modal data respectively; The feature interaction module is configured to: utilize the cross-modal interaction network obtained through federated learning, using the prompt vector as an intermediary, to perform multi-layer bidirectional interaction between the initial features of the two modalities, and obtain the final features of the two modalities after the cross-modal interaction; The similarity calculation module is configured to: calculate feature similarity based on the final features of the two modalities and screen similar second modal data; Among them, the cross-modal interaction network consists of multiple prompt layers, each prompt layer has multiple prompt vectors. Through multi-level prompt learning, the feature expression is refined in a hierarchical iterative manner, and the deep semantic association between the two modal features is gradually explored. The layer-by-layer cross-modal attention mechanism is used for calculation to integrate the semantic information of the other modality into the feature representation of the current modality to achieve cross-modal interaction.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the federated cross-modal retrieval method based on interactive prompts as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the federated cross-modal retrieval method based on interactive prompts as described in any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement a federated cross-modal retrieval method based on interactive prompts as described in any one of claims 1-6.

Images