Cross-media retrieval method and system based on adaptive prompt and implicit knowledge enhancement

Abstract

The invention belongs to the technical field of multimedia information retrieval, and particularly relates to a cross-media retrieval method and system based on adaptive prompt and implicit knowledge enhancement. The method comprises the following steps that S100, a cross-media retrieval data set is established, and the cross-media retrieval data set comprises data of two different media types; step S200, training a cross-media retrieval network based on adaptive prompt and implicit knowledge enhancement by using the cross-media retrieval data set, wherein the cross-media retrieval network comprises a potential excitation mechanism and a prompt pool mechanism based on instance-level query; and step S300, performing a cross-media retrieval test by using the trained cross-media retrieval network to realize matching between different media. Through a potential excitation mechanism and a prompt pool mechanism based on instance-level query, alignment errors caused by asymmetry of information between media and diversity of image distribution can be flexibly dealt with, so that the accuracy and robustness of cross-media retrieval are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimedia information retrieval, and in particular relates to a cross-media retrieval method and system based on adaptive prompting and implicit knowledge enhancement. Background Art

[0002] Retrieval is a key capability for information acquisition and knowledge discovery, and cross-media retrieval is an important form in the field of modern information retrieval. The goal of cross-media retrieval is to retrieve data from another media (such as images) based on input from one media (such as text). It has a wide range of application value in applications such as search engines and recommendation systems. Existing cross-media retrieval methods mainly focus on achieving similarity measurement between media through a shared semantic space, with the emphasis on semantic alignment of data of different modalities. However, the human cognitive process does not rely solely on a single media. The joint action of multiple senses such as vision and language is the basis of the cognitive world. Therefore, it is difficult to fully explore the complex associations and complementary information between different media data by relying solely on simple modality alignment methods, thus limiting the accuracy and flexibility of cross-media retrieval. Therefore, how to design a retrieval model that can fully utilize the semantic associations of multimedia data has become a key issue in improving the performance of cross-media retrieval.

[0003] Related technologies mainly include cross-media retrieval and prompt fine-tuning of large-scale pre-trained models. The goal of cross-media retrieval is to retrieve data from another media (such as images) based on data from one media (such as text) to solve the problem of aligning heterogeneous information across media. Traditional methods usually use linear mapping, such as canonical correlation analysis (CCA), to achieve similarity measurement by mapping data from different media into the same semantic space. However, due to the heterogeneity and complexity between cross-media data, linear methods are insufficient in dealing with complex scenes. In order to overcome the limitations of linear models, deep learning methods have been widely used in cross-media retrieval. For example, Rasiwasia et al. proposed a high-level semantic mapping method in the paper "A New Approach to Cross-Modal Multimedia Retrieval", which uses canonical correlation analysis to map images and texts to the same semantic space, and implements semantic learning through category labeling and logistic regression. In addition, Ngiam et al. proposed a multimedia autoencoder method in the paper "Multimodal Deep Learning", which uses the principle of minimizing reconstruction error by inputting images and texts at the same time to learn the association between different media. However, although these methods can achieve cross-media similarity retrieval, they still have certain limitations when dealing with complex multimedia relations.

[0004] In recent years, with the rapid development of pre-trained large models, how to fine-tune these models through prompts to better adapt to downstream tasks has become a hot research issue. Especially in cross-media retrieval tasks, pre-trained large models have demonstrated strong generalization capabilities. Taking CLIP as an example, this type of pre-trained model achieves cross-media semantic alignment through comparative learning of large-scale image and text pairing data, and has achieved significant performance improvements in multiple cross-media tasks. However, existing prompt-based fine-tuning methods usually use fixed-length prompts, which are difficult to flexibly adapt to the complexity of different media data, especially when processing images with wide distribution and diverse features, showing obvious limitations. In addition, existing methods often ignore the influence of information asymmetry between cross-media data, resulting in the introduction of incorrect mapping relationships in the alignment process, causing alignment errors.

[0005] Therefore, whether it is possible to provide a cross-media retrieval method and system that can flexibly cope with the asymmetry of information between media and the alignment errors caused by the diversity of image distribution, and improve the accuracy and robustness of cross-media retrieval is a technical problem that needs to be urgently solved in the present invention. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a cross-media retrieval method and system based on adaptive prompts and implicit knowledge enhancement. Through the potential stimulation mechanism and the prompt pool mechanism based on instance-level query, the present invention can flexibly deal with the asymmetry of information between media and the alignment error caused by the diversity of image distribution, thereby significantly improving the accuracy and robustness of cross-media retrieval.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The cross-media retrieval method based on adaptive hints and implicit knowledge enhancement includes the following steps:

[0009] Step S100, establishing a cross-media retrieval dataset, wherein the cross-media retrieval dataset includes data of two different media types;

[0010] Step S200, using the cross-media retrieval dataset to train a cross-media retrieval network based on adaptive prompts and implicit knowledge enhancement, wherein the cross-media retrieval network includes a potential excitation mechanism and a prompt pool mechanism based on instance-level queries;

[0011] Step S300: Perform a cross-media retrieval test using the trained cross-media retrieval network to achieve matching between different media.

[0012] Furthermore, in step S100, the data of the two different media types appear in pairs, including image data and text data; when Data is used to represent the cross-media retrieval data set, Data = {(P(I) n ,P(T) n )} N n＝1 , where N is the number of data, (P(I) n ,P(T) n ) constitutes the nth image-text pair, P(I) n represents the image, P(T) n Represents text.

[0013] Furthermore, in step S200, the training step of the cross-media retrieval network includes:

[0014] Step S210, generating a multi-angle text description for the image through a potential stimulation mechanism;

[0015] Step S220, based on the instance-level query prompt pool mechanism, dynamically query the prompt pool according to the image features to obtain the top N most relevant prompts, so as to guide the model to focus on the key parts of the image;

[0016] Step S230, fully combining the multi-angle text description and related prompt information in the shared feature space, mining the association between graphics and text, and achieving cross-media alignment.

[0017] Furthermore, in step S210, the step of generating a multi-angle text description for the image through the potential stimulation mechanism includes: using a multimodal generative pre-trained large model to input the image P(I) n Generate a multi-angle text description, the multi-angle text description includes a global perspective text description T g , Background perspective text description T b and entity perspective text description T g ; The potential excitation mechanism is used to generate multi-angle text descriptions as follows:

[0018] T g =BLIP2(Q g , P(I) n ),

[0019] T b =BLIP2(Q b , P(I) n ),

[0020] T e =BLIP2(Q e , P(I) n ),

[0021] Among them, the function BLIP2(Q,P(I) n ) represents the multimodal generative pre-trained large model BLIP2 according to the question Q for the image P(I) n Generate text description; the generated multi-angle text description is represented in the form of a set as {T g , T b , T e}, Q g , Q b , Q e represent questions based on the global, background, and entity levels, respectively; among them, Q g The global question is “Please describe the global information of the image”, Q b Question for background: "Please describe the background information of the image", Q e The question for the entity is “Please describe the image entity information”, and the text description of each perspective provides different levels of semantic information for the model.

[0022] Furthermore, in step S210, the image P(I) n The corresponding original text description P(T) n and the generated text descriptions at different angles {T g , T b , T e}The text feature set is obtained by pre-training CLIP text encoder The image P(I) n Image features obtained by pre-training CLIP image encoder

[0023] Furthermore, in step S220, the purpose of the prompt pool mechanism based on instance-level query is to dynamically generate the most appropriate prompt for each image, reflect the key visual features of the image, guide the image encoder of CLIP to focus on the key parts, and alleviate the limitation of a fixed number of prompts when facing the diversity of image data, thereby improving the accuracy of retrieval; the specific steps of the prompt pool mechanism based on instance-level query include:

[0024] Step S221, define a prompt pool:

[0025] P=[P1,P2,…,P M ],M≠0

[0026] Each of these tips is a length of L p , embedding dimension D and image embedding i emb Vectors of the same dimension, the prompt pool contains a total of M prompts;

[0027] In step S222, each prompt in the prompt pool is associated with a learnable key that allows the multimodal generative pre-trained large model to match and select the most appropriate prompt based on the input image features:

[0028] {(k1,P1),(k2,P2)…(k M ,P M )}

[0029] where k n ∈R D Yes and prompt P n Related keys, representing the key set as In order to enable the input instance to decide which hint to choose through query-key matching, a query function q∈R is introduced H×w×c →R D , maps the input image features to the same dimension as the key, and directly uses the pre-trained CLIP image encoder as the query function, taking the output of the image at the last layer as a query vector;

[0030] Step S223, by Calculate the matching score between the query vector and the prompt key, where the function Cosine_Sim( ) represents the cosine similarity between the query vector and the prompt key, and select the top N prompts with the highest scores based on the score:

[0031]

[0032] in Index a subset of N cues selected from the cue pool, and then adjust the input embedding of the image as:

[0033]

[0034] Where P sn are the prompts corresponding to the selected N prompt indices.

[0035] Furthermore, in step S300, the cross-media retrieval result includes multiple candidate items; the method for performing cross-media retrieval in step S300 includes: introducing the input image and text into the cross-media retrieval network at the same time; in the cross-media retrieval network, selecting appropriate prompts based on the prompt pool mechanism based on instance-level queries according to the input image and text, focusing on key parts of the image, and calculating the similarity between the image and the text; through the similarity score, the network generates a retrieval result for each candidate item, and outputs the text or image with the highest similarity as the final retrieval result.

[0036] A cross-media retrieval system based on adaptive hints and implicit knowledge enhancement, including:

[0037] The dataset construction module is used to construct a cross-media dataset containing image-text pairs;

[0038] A network training module, used for training a cross-media retrieval network structure based on adaptive prompts and implicit knowledge enhancement using the cross-media dataset, including fine-tuning of CLIP and training of a prompt pool module based on instance-level queries;

[0039] The retrieval module is used to perform cross-media retrieval through a trained cross-media retrieval network based on adaptive prompts and implicit knowledge enhancement, so as to achieve effective retrieval between text and image media.

[0040] The beneficial effects of the present invention are:

[0041] The present invention is used for accurate alignment and efficient retrieval of different media data (such as text and images). Compared with the prior art, it can effectively solve the alignment error caused by information asymmetry between media and diversity of image distribution; through the potential stimulation mechanism, it supplements the information asymmetry between the original text and the image; through the instance-level query prompt pool, prompts are dynamically selected, which alleviates the limitation of a fixed number of prompts in the face of image distribution diversity, thereby improving the accuracy and robustness of cross-media retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the cross-media retrieval method based on adaptive prompts and implicit knowledge enhancement of the present invention.

[0043] Figure 2 It is a schematic diagram of the cross-media retrieval network structure of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Example 1

[0046] A cross-media retrieval method based on adaptive hints and implicit knowledge enhancement, such as Figure 1 As shown, the following steps are included.

[0047] Step S100, establish a cross-media retrieval dataset, the cross-media retrieval dataset includes data of two different media types. The two different media types of data appear in pairs, including image data and text data. Data represents the cross-media retrieval dataset, Data = {(P(I) n ,P(T) n )} N n＝1 , where N is the number of data, (P(I) n ,P(T) n) constitutes the nth image-text pair, P(I) n represents the image, P(T) n Represents text.

[0048] Step S200, the network structure of this step is as follows Figure 2 The cross-media retrieval dataset is used to train a cross-media retrieval network based on adaptive prompts and implicit knowledge enhancement, and the cross-media retrieval network includes a potential excitation mechanism and a prompt pool mechanism based on instance-level queries.

[0049] In step S200, the training steps of the cross-media retrieval network include: step S210, generating a multi-angle text description for the image through a potential stimulation mechanism; step S220, based on an instance-level query prompt pool mechanism, dynamically querying the top N most relevant prompts from the prompt pool according to image features to guide the model to focus on the key parts of the image; step S230, fully combining the multi-angle text description and related prompt information in a shared feature space, mining the association between graphics and text, and realizing cross-media alignment.

[0050] In this embodiment, since text is an abstract representation of an image and cannot cover all the semantic information of the image, an incorrect mapping relationship will be introduced in the process of aligning the image and text. In order to alleviate the information asymmetry between the media, a potential excitation mechanism is proposed. Specifically, in step S210, a multi-angle text description is generated for the image through the potential excitation mechanism: the multi-angle questioning method is used to make the multimodal generative pre-trained large model generate a multi-angle text description for the input image P(I) n Generate a multi-angle text description. The multi-angle text description includes a global perspective text description T g , Background perspective text description T b and entity perspective text description T g。 The potential excitation mechanism is used to generate multi-angle text descriptions as follows:

[0051] T g =BLIP2(Q g , P(I) n ),

[0052] T b =BLIP2(Q b , P(I) n ),

[0053] T e =BLIP2(Q e , P(I) n ),

[0054] Among them, the function BLIP2(Q,P(I) n) represents the multimodal generative pre-trained large model BLIP2 according to the question Q for the image P(I) n Generate text description; the generated multi-angle text description is represented in the form of a set as {T g , T b , T e}, Q g , Q b , Q e represent questions based on the global, background, and entity levels, respectively; among them, Q g The global question is “Please describe the global information of the image”, Q b Question for background: "Please describe the background information of the image", Q e The question for the entity is "Please describe the image entity information". The text description of each perspective provides different levels of semantic information for the model, thereby enriching the semantic representation of text media and improving the coverage of text to image content. n The corresponding original text description P(T) n and the generated text descriptions at different angles {T g , T b , T e}The text feature set is obtained by pre-training CLIP text encoder The image P(I) n Image features obtained by pre-training CLIP image encoder

[0055] In step S220, the purpose of the prompt pool mechanism based on instance-level query is to dynamically generate the most appropriate prompt for each image, reflect the key visual features of the image, guide the image encoder of CLIP to focus on the key parts, and alleviate the limitations of a fixed number of prompts when facing the diversity of image data, thereby improving the accuracy of retrieval. The specific steps of the prompt pool mechanism based on instance-level query include steps S221-step S223.

[0056] Step S221, the prompt pool is defined as:

[0057] P=[P1,P2,…,P M ],M≠0

[0058] Each of these tips is a length of L p , embedding dimension D and image embedding i emb A vector of the same dimension, with a total of M prompts in the prompt pool.

[0059] In step S222, each prompt in the prompt pool is associated with a learnable key that allows the multimodal generative pre-trained large model to match and select the most appropriate prompt based on the input image features:

[0060] {(k1,P1),(k2,P2)…(k M ,P M )}

[0061] where k n ∈R D Yes and prompt P n Related keys, representing the key set as In order to enable the input instance to decide which hint to choose through query-key matching, a query function q∈R is introduced H×w×c →R D , maps the input image features to the same dimension as the key, and directly uses the pre-trained CLIP image encoder as the query function, taking the output of the image at the last layer as the query vector.

[0062] Step S223, by Calculate the matching score between the query vector and the prompt key, where the function Cosine_Sim( ) represents the cosine similarity between the query vector and the prompt key, and select the top N prompts with the highest scores based on the score:

[0063]

[0064] in Index a subset of N cues selected from the cue pool, and then adjust the input embedding of the image as:

[0065]

[0066] in are the prompts corresponding to the selected N prompt indices.

[0067] Step S300, using the trained cross-media retrieval network to perform a cross-media retrieval test to achieve retrieval matching between different media. The cross-media retrieval result includes multiple candidate items.

[0068] After the training is completed, the trained cross-media retrieval network is used for testing. The method for cross-media retrieval in step S300 includes: introducing the input image and text into the cross-media retrieval network at the same time; in the cross-media retrieval network, selecting appropriate prompts through a prompt pool mechanism according to the input image and text, focusing on the key parts of the image, and calculating the similarity between the image and the text; through the similarity score, the network generates a retrieval result for each candidate, and outputs the text or image with the highest similarity as the final retrieval result.

[0069] The following experimental results show that compared with the existing methods, the cross-media retrieval method based on adaptive prompts and implicit knowledge enhancement of the present invention can achieve higher retrieval accuracy.

[0070] In this embodiment, experiments are conducted based on the cross-media datasets Flickr30K and MSCOCO.

[0071] The Flickr30K dataset was proposed by the paper "From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference over Event Descriptions" (authors: Peter Young, Alice Lai, Micah Hodosh and Julia Hockenmaier, published in Transactions of the Association for Computational Linguistics in 2014), containing 31,783 images, each of which contains 5 text descriptions.

[0072] The MSCOCO dataset was proposed by the paper "Microsoft COCO: Common Objects in Context" (authors: Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár and C. Lawrence Zitnick, published at the European Conference on Computer Vision in 2014), and contains 123,287 images, each of which has 5 text descriptions. In this example, the following three methods are mainly tested for experimental comparison:

[0073] Existing method 1: The SGRAF method in the paper “Similarity Reasoning and Filtration for Image-Text Matching” (by Haiwen Diao, Ying Zhang, Lin Ma and Huchuan Lu) uses the similarity graph reasoning (SGR) and similarity attention filtering (SAF) modules to reason about global and local relational similarities and filter out irrelevant alignments, respectively, to improve the accuracy and performance of image-text matching.

[0074] Existing method 2: The NAAF method in the document “Negative Aware Attention Framework for Image-Text Matching.” (authors Kun Zhang, Zhendong Mao, Quan Wang and Yongdong Zhang;) calculates the image-text similarity more accurately by simultaneously utilizing the positive and negative influences of matching and non-matching fragments, thereby improving the accuracy of cross-media retrieval.

[0075] Existing method three: The MGCN method in the document "MKVSE: Multimodal Knowledge Enhanced Visual-semantic Embedding for Image-text Retrieval." (Author: Duoduo Feng, Xiangteng He, Yuxin Peng) introduces multimedia knowledge graphs, explicitly expresses implicit multimedia knowledge relationships, and injects them into visual-semantic embedding to improve the accuracy of cross-media retrieval.

[0076] In the embodiment, the recall rate (Recall@k) indicator is used to evaluate the effect of cross-media retrieval. The recall rate refers to the ratio of the relevant data retrieved by the model in the first k search results to all the relevant data in the data set. The higher the Recall@k, the more relevant data the model retrieves in the first k search results, indicating that the model performs better in cross-media retrieval.

[0077] Table 1. Experimental results of the present invention on the Flickr30K dataset

[0078]

[0079] Table 2. Experimental results of the present invention on the MSCOCO dataset

[0080]

[0081]

[0082] As can be seen from Tables 1 and 2, the present invention has achieved better results than existing methods in terms of the total recall rate (RSUM) of cross-media retrieval. Existing method one (SGRAF) improves global and local similarity alignment through similarity graph reasoning and attention filtering modules, and achieves good results, but this method does not make sufficient use of the information of unmatched segments and fails to fully mine the negative features hidden in cross-media data. Existing method two (NAAF) emphasizes the importance of unmatched segments and makes detailed distinctions and inferences between matching and unmatched segments through a negative-aware attention framework, but due to the lack of dynamic adjustment of the prompt mechanism, its retrieval robustness still has room for improvement. Existing method three (MKVSE) adopts a visual-semantic embedding method enhanced by multimedia knowledge, and enhances cross-media matching by introducing a multimodal knowledge graph, but its processing method is limited to static semantic enhancement and lacks a dynamic adjustment mechanism, so the retrieval effect in complex scenarios is limited.

[0083] In contrast, the present invention significantly improves the alignment effect of images and texts through the instance-level query prompt pool mechanism and the potential stimulation mechanism. On the one hand, the instance-level query prompt pool mechanism can dynamically select the prompts most relevant to the image according to the specific input, thereby effectively coping with the high distribution diversity within the media; on the other hand, the potential stimulation mechanism further enriches the fine-grained information of cross-media alignment by generating multi-angle text descriptions, alleviates the asymmetry of information between different media, and reduces the occurrence of incorrect mapping relationships during training. Through these two core mechanisms, the present invention effectively integrates the complementary information in images and texts on the one hand, and improves the accuracy and robustness of cross-media retrieval through dynamic adjustment strategies on the other hand, surpassing the cross-media retrieval performance of existing methods on the Flickr30K and MSCOCO datasets.

[0084] Example 2

[0085] Based on the same inventive concept, another embodiment of the present invention provides a cross-media retrieval system based on adaptive prompting and implicit knowledge enhancement, which includes:

[0086] The dataset construction module is used to construct a cross-media dataset containing image-text pairs;

[0087] A network training module for training a cross-media retrieval network structure based on adaptive prompts and implicit knowledge enhancement using a cross-media dataset of image-text pairs, including fine-tuning of CLIP and training of a prompt pooling module based on instance-level queries;

[0088] The retrieval module is used to perform cross-media retrieval through a trained cross-media retrieval network based on adaptive prompts and implicit knowledge enhancement, so as to achieve effective retrieval between text and image media.

[0089] Example 3

[0090] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the processor is used to execute the computer program stored in the memory, and the computer program includes instructions for each step in the cross-media retrieval method based on adaptive prompts and implicit knowledge enhancement of the present invention.

[0091] Example 4

[0092] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium including ROM / RAM, a disk or an optical disk, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the various steps of the method of the present invention are implemented.

[0093] In this embodiment, the media types are text and image. The method of the present invention also supports mutual retrieval tasks of other media types, including image and audio, text and audio, etc.

[0094] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A cross-media retrieval method based on adaptive hints and implicit knowledge enhancement, characterized in that: The following steps are involved: Step S100, establishing a cross-media retrieval dataset, wherein the cross-media retrieval dataset includes data of two different media types; Step S200, using the cross-media retrieval dataset to train a cross-media retrieval network based on adaptive prompts and implicit knowledge enhancement, wherein the cross-media retrieval network includes a potential excitation mechanism and a prompt pool mechanism based on instance-level queries; Step S300: Perform a cross-media retrieval test using the trained cross-media retrieval network to achieve matching between different media.

2. The cross-media retrieval method based on adaptive prompting and implicit knowledge enhancement according to claim 1, characterized in that: In step S100, the data of the two different media types appear in pairs, including image data and text data; when Data is used to represent the cross-media retrieval data set, Data = {(P(I) n ,P(T) n )} N n＝1 , where N is the number of data, (P(I) n ,P(T) n ) constitutes the nth image-text pair, P(I) n represents the image, P(T) n Represents text.

3. The cross-media retrieval method based on adaptive prompting and implicit knowledge enhancement according to claim 1, characterized in that: In step S200, the training step of the cross-media retrieval network includes: Step S210, generating a multi-angle text description for the image through a potential stimulation mechanism; Step S220, based on the instance-level query prompt pool mechanism, dynamically query the prompt pool according to the image features to obtain the top N most relevant prompts, so as to guide the model to focus on the key parts of the image; Step S230, fully combining the multi-angle text description and related prompt information in the shared feature space, mining the association between graphics and text, and achieving cross-media alignment.

4. The cross-media retrieval method based on adaptive prompting and implicit knowledge enhancement according to claim 3 is characterized in that: In step S210, the step of generating a multi-angle text description for the image through the potential stimulation mechanism includes: using a multimodal generative pre-trained large model to input the image P(I) n Generate a multi-angle text description, the multi-angle text description includes a global perspective text description T g , Background perspective text description T b and entity perspective text description T g ; The potential excitation mechanism is used to generate multi-angle text descriptions as follows: T g =BLIP2(Q g ,P(I) n ), T b =BLIP2(Q b ,P(I) n ), T e =BLIP2(Q e ,P(I) n ), Among them, the function BLIP2(Q,P(I) n ) represents the multimodal generative pre-trained large model BLIP2 according to the question Q for the image P(I) n Generate text description; the generated multi-angle text description is represented in the form of a set as {T g , T b , T e }, Q g , Q b , Q e represent questions based on the global, background, and entity levels, respectively; among them, Q g The global question is "Please describe the global information of the image", Q b Question for background "Please describe the background information of the image", Q e The question for the entity is "Please describe the image entity information", and the text description of each perspective provides different levels of semantic information for the model.

5. The cross-media retrieval method based on adaptive prompting and implicit knowledge enhancement according to claim 4, characterized in that: In step S210, the image P(I) n The corresponding original text description P(T) n and the generated text descriptions at different angles {T g , T b , T e }The text feature set is obtained by pre-training CLIP text encoder The image P(I) n Image features obtained by pre-training CLIP image encoder 6. The cross-media retrieval method based on adaptive prompting and implicit knowledge enhancement according to claim 3 is characterized in that: In step S220, the purpose of the prompt pool mechanism based on instance-level query is to dynamically generate the most appropriate prompt for each image, reflecting the key visual features of the image, guiding the image encoder of CLIP to focus on the key parts, and alleviating the limitation of a fixed number of prompts when facing the diversity of image data, thereby improving the accuracy of retrieval; The specific steps of the hint pool mechanism based on instance-level query include: Step S221, define a prompt pool: P=[P1,P2,…,P M ],M≠0 Each of these tips is a length of L p , embedding dimension D and image embedding i emb Vectors of the same dimension, the prompt pool contains a total of M prompts; In step S222, each prompt in the prompt pool is associated with a learnable key that allows the multimodal generative pre-trained large model to match and select the most appropriate prompt based on the input image features: {(k1,P1),(k2,P2)…(k M ,P M )} where k n ∈R D Yes and prompt P n Related keys, representing the key set as In order to enable the input instance to decide which hint to choose through query-key matching, a query function q∈R is introduced H×w×c →R D , maps the input image features to the same dimension as the key, and directly uses the pre-trained CLIP image encoder as the query function, taking the output of the image at the last layer as a query vector; Step S223, by Calculate the matching score between the query vector and the prompt key, where the function Cosine_Sim( ) represents the cosine similarity between the query vector and the prompt key, and select the top N prompts with the highest scores based on the score: in Index a subset of N cues selected from the cue pool, and then adjust the input embedding of the image as: in are the prompts corresponding to the selected N prompt indices.

7. The cross-media retrieval method based on adaptive prompting and implicit knowledge enhancement according to claim 1, characterized in that: In step S300, the cross-media retrieval result includes multiple candidate items; the method for performing cross-media retrieval in step S300 includes: introducing the input image and text into the cross-media retrieval network at the same time; in the cross-media retrieval network, selecting appropriate prompts based on the prompt pool mechanism based on instance-level queries according to the input image and text, focusing on key parts of the image, and calculating the similarity between the image and the text; through the similarity score, the network generates a retrieval result for each candidate item, and outputs the text or image with the highest similarity as the final retrieval result.

8. A cross-media retrieval system based on adaptive hints and implicit knowledge enhancement, characterized in that: include: The dataset construction module is used to construct a cross-media dataset containing image-text pairs; A network training module, used for training a cross-media retrieval network structure based on adaptive prompts and implicit knowledge enhancement using the cross-media dataset, including fine-tuning of CLIP and training of a prompt pool module based on instance-level queries; The retrieval module is used to perform cross-media retrieval through a trained cross-media retrieval network based on adaptive prompts and implicit knowledge enhancement, so as to achieve effective retrieval between text and image media.

Images