Cross-modal hash retrieval method based on prompt learning and adaptive Mama gating selection fusion

Abstract

The invention discloses a cross-modal Hash retrieval method based on prompt learning and adaptive Mama gating selection fusion, and mainly relates to the field of deep cross-modal Hash retrieval. Comprising the following steps: S1, respectively and correspondingly dividing an image data set and category label information corresponding to the image data set into a training set and a retrieval set; s2, preprocessing the image information; s3, constructing a cross-modal Hash retrieval framework based on prompt learning and adaptive Mamba gating selection fusion; s4, obtaining a model through training by using the training set; s5, inputting the images of the retrieval set into the trained model; s6, storing the best model weight of the test set, and storing the hash code with the best performance effect on the test set; according to the method, the occurrence of semantic truncation can be relieved, the deficiency of context semantics can be supplemented, and different modal features can be reselected and fused, so that the semantic feature weight beneficial to a retrieval result is improved, and the weight of negative semantic information is reduced.

Description

Technical Field

[0001] The present invention relates to the field of deep cross-modal hash retrieval, and in particular to a cross-modal hash retrieval method based on the fusion of prompt learning and adaptive Mamba gating selection. Background Art

[0002] In recent years, with the rapid development of smart devices and social media, Internet data has taken on a variety of forms, including text, images and other modalities. These multimodal data have accumulated a huge amount of information resources in the fields of Internet technology, multimedia retrieval and social networks. However, effectively organizing, retrieving and understanding these different modal data has become a major challenge for academia and industry. In order to achieve efficient cross-modal retrieval, hashing methods are widely used in the indexing and retrieval process of cross-modal data. Cross-modal hash retrieval learns a shared hash function, where nonlinear hash functions have strong representation capabilities and can map data of different modalities into a common hash space.

[0003] Generally speaking, existing cross-modal hashing methods can be roughly divided into unsupervised cross-modal hashing and supervised cross-modal hashing. Compared with unsupervised methods, supervised cross-modal hashing methods usually have better performance. Although deep supervised cross-modal hashing has made extensive progress, there are still some problems: taking the MS_COCO dataset commonly used in cross-modal hashing as an example, the text length can reach up to 625 characters at most and 169 characters at least, which is far more than the 32 characters that the text encoder of the CLIP pre-trained model can accommodate, thus leading to serious text semantic truncation problems. Therefore, how to alleviate the occurrence of this phenomenon and supplement the lack of contextual semantics has become a major difficulty; in order to alleviate this problem, previous methods use the bag-of-words BOW model to encode text Encoding, at the same time, this method ignores the relevance of context words, resulting in the inability to improve retrieval performance. Existing methods use pre-trained Transformer models such as BERT to process text, which to a certain extent alleviates the situation of irrelevance of context information caused by BOW encoding, but there will be text truncation problems. The mPLUG large language model is used to reconstruct text as a prompt, and at the same time, visual and text prompting patterns are introduced. However, the use of a large language model for reconstruction will bring a significant increase in the amount of calculation and the addition of other models to assist. Although the retrieval performance is improved, the use of a large language model to optimize text does not solve the problem of truncation of the original text semantics. These problems make it very challenging to converge to the global optimum, and ultimately produce a suboptimal hash code. Therefore, how to increase the weight of important features in the truncated text while retaining contextual information has become a hot topic of research. Existing cross-modal hashing methods further learn the consistency between different modalities by introducing contrastive learning. Due to the challenges of collecting real-world data from noisy websites and the expensive annotation process, the benchmark datasets used in the field of cross-modal retrieval, including MIRFLICKR-25K, NUS-WIDE, and MS-COCO, still have inherent limitations caused by context loss and information redundancy, especially in the representation of text data. The text content in the MIRFLICKR-25K and NUS-WIDE datasets is generated by the direct connection of multiple labels, resulting in a lack of contextual information. In addition, the text content in the MS-COCO dataset consists of merging multiple manually annotated image descriptions, which may introduce a certain degree of semantic redundancy. These limitations cause the network model to inevitably introduce contextual redundant information when processing different modal information, resulting in the generation of suboptimal binary hash codes and reducing retrieval performance. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and to provide a cross-modal hash retrieval method based on prompt learning and adaptive Mamba gating selection fusion, which can alleviate the occurrence of semantic truncation, supplement the lack of contextual semantics, and reselect and fuse different modal features to increase the weight of semantic features that are beneficial to the retrieval results and reduce the weight of negative semantic information.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] A cross-modal hash retrieval method based on hint learning and adaptive Mamba gated selection fusion includes the following steps:

[0007] S1. The image dataset and the category label information corresponding to the image dataset are divided into a training set and a retrieval set, where each sample includes an image and a corresponding text description. The training set and the retrieval set have the same number of samples for each category. On this basis, a dynamic and learnable prompt word is generated for each image-text pair.

[0008] S2, preprocess the image information, make the input image resolution the same, and apply random flipping and random cropping to enhance the robustness of the network;

[0009] S3, build a cross-modal hash retrieval architecture based on prompt learning and adaptive Mamba gated selection fusion, which includes CLIP feature extraction backbone and adaptive Mamba gated selection fusion module, where CLIP feature extraction backbone is used to learn image and text representation, and Mamba gated selection fusion module is used to select and filter image and text features;

[0010] S4, using the training set, according to the cross-modal hash retrieval architecture based on prompt learning and adaptive Mamba gating selection fusion, a cross-modal hash retrieval model based on prompt learning and adaptive Mamba gating selection fusion is obtained through training;

[0011] S5, input the retrieval set images into the trained cross-modal hash retrieval model based on prompt learning and adaptive Mamba gated selection, obtain the hash code of the retrieval set, and adjust the model parameters through the back propagation algorithm;

[0012] S6. Take the best model weight tested on the test set and save it, and save the hash code that performs best on the test set.

[0013] Preferably, in step S3, the CLIP framework is used as the feature extraction network in the cross-modal hash retrieval architecture based on prompt learning and adaptive Mamba gating selection, wherein the image feature extraction module adopts the Vision Transformer network, and the text feature extraction module selects the GPT2 network and uses the ViT-B-16 pre-trained weights.

[0014] Preferably, in step S3: in a cross-modal hash retrieval architecture based on cue learning and adaptive Mamba gating selection fusion, a two-branch Transformer is used to extract image and text features; a Transformer network for extracting cue features is used to extract cue weight similarity.

[0015] Preferably, in step S3: in the cross-modal hash retrieval architecture based on hint learning and adaptive Mamba gated selection, the pre-trained ViT is used as the visual encoder to extract the image-text representation Use pre-trained GPT2 as a text encoder to extract image and text representation The constructed mask prompt sentence tokens are embedded into the feature representation as A two-layer Transformer is used to construct the prompt learning module, in which the prompt encoder is responsible for adapting and fusing the prompt features with the original text features.

[0016] Preferably, in step S3: the cross-modal hash retrieval architecture based on hint learning and adaptive Mamba gated selection fusion includes a text affinity guidance hint module, which can dynamically learn and distinguish foreground information that is helpful for retrieval, and optimize text feature representation through a dynamic hint mechanism;

[0017] There is a category label set l i Corresponding to the image-text pair i and t , the generation of prompt text includes the following steps: guiding prompt generation through the initial context template, which is in the form of: This is an image containing{class_name i}, where {class_name i} represents label l i The category name is obtained by combining the context template with the category tag name class_name i Combine to generate a complete prompt word:

[0018] prompt i ="This is an image containing{class_name i}"

[0019] After constructing the prompt word, use the tokenizer to tokenize the prompt word prompt i Perform BPE encoding to obtain the token sequence:

[0020] token_prompti =Tokenizer(prompt i )

[0021] By encoding the token sequence, we get the embedding vector p of the prompt word i :

[0022]

[0023] where d p is the embedding dimension of CLIP;

[0024] Finally, the resulting cue embedding is combined with the learnable context vector C ctx Combined to form the final prompt text features:

[0025] f i p =p i +C ctx

[0026] Among them C ctx It is a learnable context vector. By dynamically generating a learnable Mask mask corresponding to the prompt word, it can help the network dynamically learn to distinguish the feature information in the image and text that is beneficial to the retrieval results, dynamically distinguish the foreground and background information in the image, and use the dynamic Mask mechanism to dynamically cover the background information in the text, retain the foreground features, and at the same time, the learned dynamic prompts are weighted with the original text to reduce the weight of the features in the original text that are irrelevant to the retrieval results and increase the weight ratio of the features that are beneficial to the retrieval; the generated prompt text f i p As the input of the TAP module; the TAP module is mainly composed of two layers of Transformer encoders, which are responsible for fusing the prompt text and the original text features; through the Transformer layer in the TAP module, the prompt text features f i p and the original text features f i t , which is expressed as:

[0027] f i pt = TransformerLayer(f i t ,f i p )

[0028] The fused prompt text feature matrix And the original text feature matrix Perform element-by-element multiplication and obtain weighted feature representation through global average pooling operation:

[0029]

[0030] Among them, AvgPool represents the global average pooling, f i pt is the final weighted text feature representation.

[0031] Preferably, in step S3: firstly, the image feature and the obtained mixed prompt text feature are filtered and the cross-modal feature is fused to obtain the fused feature, and the following formula can be obtained:

[0032] f i fusion =Concat(SiLU(Mlp(LN(f i v ))),SiLU(Mlp(LN(f i t ))))

[0033] Among them, Concat represents concatenation operation, LN represents regularization operation, Mlp represents multi-layer perceptron, and SiLU represents activation function;

[0034] Then the fused features are sent to the Mamba adaptive selection module, namely:

[0035]

[0036] Among them, Flip axis＝i ,i∈{1,2} indicates flipping along a specific axis. T represents the reverse operation of flipping, SSM represents the state space model;

[0037] Then, the three features obtained are subjected to adaptive weighted filtering as follows:

[0038]

[0039] Among them, temp is the learnable temperature coefficient, which is used to balance the weights of the four features before and after flipping, and θ represents the learnable parameter;

[0040] Finally, the filtered features are passed through a layer of Transformer encoder to obtain:

[0041]

[0042] The obtained features It represents the features obtained by adaptive Mamba gating selection fusion module, and Split represents the separation of individual image and text features from the fusion features.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] The present invention proposes a cross-modal hash retrieval method based on hint learning and Mamba adaptive gating selection fusion. In order to alleviate the occurrence of semantic truncation and supplement the lack of contextual semantics, the present invention introduces adaptive text hints to assist the learning of truncated text features, so that the weight of the features in the semantics of the truncated text that are beneficial to the retrieval results can be more highlighted, and the retrieval results can be optimized. In view of the limitations of upper and lower redundancy between different modal information, the present invention designs a Mamba-based adaptive gating selection fusion mechanism to reselect and fuse different modal features, so as to increase the weight of semantic features that are beneficial to the retrieval results and reduce the weight of negative semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall framework diagram of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0047] Embodiment: The present invention is a cross-modal hash retrieval method based on hint learning and adaptive Mamba gated selection fusion, characterized in that it includes the steps of:

[0048] S1. Divide the image dataset and its corresponding category label information into two parts: a training set and a retrieval set. Each sample includes an image and a corresponding text description. The training set and the retrieval set have the same number of samples for each category. On this basis, generate dynamic and learnable prompt words for each image-text pair.

[0049] S2, preprocess the image information, input the image with a resolution of 224*224, and apply random flipping and random cropping to enhance the robustness of the network;

[0050] S3, build a cross-modal hash retrieval architecture based on prompt learning and adaptive Mamba gated selection fusion, which includes CLIP (using ViT-B-16 pre-trained weights) feature extraction backbone and adaptive Mamba gated selection fusion module, where CLIP feature extraction backbone is used to learn image and text representation, and Mamba gated selection fusion module is used to select and filter image and text features;

[0051] S4, using the training set, according to the cross-modal hash retrieval architecture based on prompt learning and adaptive Mamba gating selection fusion, a cross-modal hash retrieval model based on prompt learning and adaptive Mamba gating selection fusion is obtained through training;

[0052] S5, input the retrieval set images into the trained cross-modal hash retrieval model based on prompt learning and adaptive Mamba gated selection, obtain the hash code of the retrieval set, and adjust the model parameters through the back propagation algorithm;

[0053] S6. Take the best model weight tested on the test set and save it, and save the hash code that performs best on the test set.

[0054] Next, the prompt learning and adaptive Mamba gating selection fusion modules mentioned in the above steps will be introduced in detail.

[0055] The cross-modal hash retrieval architecture based on hint learning and adaptive Mamba gating selection in step 3 above is as follows Figure 1 As shown. First, the CLIP framework is used as the feature extraction network, in which the image feature extraction module adopts the VisionTransformer network, and the text feature extraction module selects the GPT2 network. Using the ViT-B-16 pre-trained model can greatly reduce the time required for training. In addition, thanks to CLIP's pre-training on large-scale unlabeled image and text pairs, the intrinsic relationship between images and texts is learned, so that modal alignment is naturally formed in the model. In addition, by calculating the similarity between image and text embeddings, positive examples (correct image-text pairs) are brought closer, while negative examples (irrelevant image-text pairs) are pulled further away, thereby generating hash codes that are not only fast in retrieval speed on large-scale datasets but also maintain high retrieval accuracy.

[0056] In this paper, a framework similar to most cross-modal hashing methods is adopted, with image-text pairs as input. Suppose there are N training sample pairs Each sample By image features Text Features Multi-label annotation i ∈{0,1} 1xC and the prompt word p i Here, d v and d t Represent the dimensions of image and text modality features respectively, and C is the total number of categories. i belongs to category c, then l i,c =1; otherwise l i,c = 0. In order to capture the semantic similarity between sample pairs, the label l i and lj Generate an N×N similarity matrix S. It is defined as: if i and j belongs to at least one of the same categories, then S ij =1, indicating that the two samples are semantically similar; otherwise, S ij = 0, indicating that they are not semantically similar. The goal of cross-modal hashing is to project high-dimensional features of different modalities into a common K-bit discrete hash space and learn a unified hash code by using the similarity matrix S for supervision. Specifically, this method needs to learn two hash functions, which act on the features of image and text modalities respectively, and map them into fixed-length hash codes. Its mathematical expression is: and in, and represents the K-bit hash code of the image and text modalities of the i-th sample, θ v and θ t are learnable parameters of the hash function. By optimizing these hash functions, the model can ensure that the learned hash codes not only maintain the similarity between image and text modalities in the hash space, but also effectively capture semantic information, so that samples of different modalities have similar representations in the shared hash space.

[0057] Tips for learning mechanisms:

[0058] This module is the core module of the present invention, which is composed of a prompt learning module. The specific algorithm is as follows:

[0059] Figure 1 The specific process of the present invention is demonstrated. In order to utilize the latest deep neural network and the latest research results based on Transformer, the present invention adopts a two-branch Transformer to extract image and text features. In addition, the present invention also designs a Transformer network for extracting prompt features for extracting prompt weight similarity.

[0060] In order to obtain better image and text features, the present invention uses pre-trained ViT and GPT2 as visual encoder and text encoder to extract image and text representation and Similarly, the feature representation of the constructed mask prompt sentence token after embedding is In the prompt encoder, the present invention adopts two layers of Transformer to construct a prompt learning module, wherein the prompt encoder is responsible for adapting and fusing the prompt features with the original text features.

[0061] In the cross-modal hashing task, negative features in the text affect the retrieval performance, resulting in the inability to effectively distinguish foreground and background information. To solve this problem, this paper proposes a Text Affinity Aware Prompt Module (TAP), which can dynamically learn and distinguish foreground information that is helpful for retrieval, and optimize the text feature representation through a dynamic prompting mechanism.

[0062] There is a category label set l i Corresponding to the image-text pair i and t , the generation of prompt text includes the following steps: guiding prompt generation through the initial context template, which is in the form of: This is an image containing{class_name i}, where {class_name i} represents label l i The category name is obtained by combining the context template with the category tag name class_name i Combine to generate a complete prompt word:

[0063] prompt i ="This is an image containing{class_name i}"

[0064] After constructing the prompt word, use the tokenizer to tokenize the prompt word prompt i Perform BPE encoding to obtain the token sequence:

[0065] token_prompt i =Tokenizer(prompt i )

[0066] By encoding the token sequence, we get the embedding vector p of the prompt word i :

[0067]

[0068] where d p is the embedding dimension of CLIP.

[0069] Finally, the resulting cue embedding is combined with the learnable context vector C ctx Combined to form the final prompt text features:

[0070] f i p =p i +C ctx

[0071] Among them C ctx It is a learnable context vector. By dynamically generating a learnable Mask corresponding to the prompt word, it can help the network dynamically learn to distinguish the feature information in the image and text that is beneficial to the retrieval results, dynamically distinguish the foreground and background information in the image, and use the dynamic Mask mechanism to dynamically cover the background information in the text, retain the foreground features, and at the same time, the learned dynamic prompts are weighted with the original text to reduce the weight of the features in the original text that are irrelevant to the retrieval results and increase the weight ratio of the features that are beneficial to the retrieval. The generated prompt text f i p As the input of the TAP module. The TAP module is mainly composed of two layers of Transformer encoders, which are responsible for fusing the prompt text and the original text features. Through the Transformer layer in the TAP module, the prompt text features f i p and the original text features f i t , which is expressed as:

[0072] f i pt = TransformerLayer(f i t ,f i p )

[0073] The fused prompt text feature matrix And the original text feature matrix Perform element-by-element multiplication and obtain weighted feature representation through global average pooling operation:

[0074]

[0075] Among them, AvgPool represents the global average pooling, f i pt is the final weighted text feature representation. Through this fusion process, useful information related to the foreground is effectively retained and the interference of background noise on the retrieval results is reduced. Through the dynamic fusion of the prompt information of the TAP module and the text features, the system can adaptively learn the foreground information, enhance the retrieval performance and reduce the interference of irrelevant features, significantly improving the effect of cross-modal hash retrieval.

[0076] Adaptive Mamba Gating Select Fusion:

[0077] Mamba is a new selective structural state-space model that excels in long sequence modeling tasks. Mamba alleviates the modeling constraints of convolutional neural networks through global receptive fields and dynamic weighting, and provides advanced modeling capabilities similar to Transformers. Crucially, it achieves this without incurring the quadratic computational complexity typically associated with Transformers. The present invention combines Mamba with adaptive gating to design an adaptive filtering feature fusion module, called the adaptive Mamba gated selective fusion module. The text features are represented by weighted text features through the above-mentioned prompt learning module. However, feature filtering and cross-modal feature fusion are also required between the image features and the obtained hybrid prompt text features. Therefore, the following formula can be obtained:

[0078] f i fusion =Concat(SiLU(Mlp(LN(f i v ))),SiLU(Mlp(LN(f i t ))))

[0079] Among them, Concat represents the concatenation operation, LN represents the regularization operation, Mlp represents the multi-layer perceptron, and SiLU represents the activation function.

[0080] After obtaining the fusion features, the present invention sends them to the Mamba adaptive selection module, namely:

[0081]

[0082] Among them, Flip axis＝i ,i∈{1,2} indicates flipping along a specific axis. T represents the reverse operation of flipping, SSM represents the state space model, and the present invention performs adaptive weighted filtering on the three features finally obtained, and the operation is as follows:

[0083]

[0084] Among them, temp is the learnable temperature coefficient, which is used to balance the weights of the four features before and after flipping, and θ represents the learnable parameter.

[0085] Finally, the present invention passes the filtered features through a layer of Transformer encoder to obtain:

[0086]

[0087] The obtained features It represents the features obtained by adaptive Mamba gating selection fusion module, and Split represents the separation of individual image and text features from the fusion features.

[0088] Through the above operation, the weight of semantic features that are beneficial to the retrieval results can be increased, and the weight of negative semantic information can be reduced. The present invention combines the advantages of both Mamba and Transformer, and further designs an adaptive gated fusion selection module on this basis, which can simultaneously select and retain semantic information that is beneficial to the retrieval in the two modes, while filtering the graphic semantic information that has a negative effect on the retrieval, that is, redundant context information. Through the above two modules, the problem of text semantic truncation and how to effectively fuse different modal features in cross-modal features can be solved at the same time, so as to increase the weight of semantic features that are beneficial to the retrieval results, reduce the weight of negative semantic information, and finally significantly improve the cross-modal retrieval effect.

Claims

1. A cross-modal hash retrieval method based on hint learning and adaptive Mamba gated selection, characterized in that: Includes steps: S1. The image dataset and the category label information corresponding to the image dataset are divided into a training set and a retrieval set, where each sample includes an image and a corresponding text description. The training set and the retrieval set have the same number of samples for each category. On this basis, a dynamic and learnable prompt word is generated for each image-text pair. S2, preprocess the image information, make the input image resolution the same, and apply random flipping and random cropping to enhance the robustness of the network; S3, build a cross-modal hash retrieval architecture based on prompt learning and adaptive Mamba gated selection fusion, which includes CLIP feature extraction backbone and adaptive Mamba gated selection fusion module, where CLIP feature extraction backbone is used to learn image and text representation, and Mamba gated selection fusion module is used to select and filter image and text features; S4, using the training set, according to the cross-modal hash retrieval architecture based on prompt learning and adaptive Mamba gating selection fusion, a cross-modal hash retrieval model based on prompt learning and adaptive Mamba gating selection fusion is obtained through training; S5, input the retrieval set images into the trained cross-modal hash retrieval model based on prompt learning and adaptive Mamba gated selection, obtain the hash code of the retrieval set, and adjust the model parameters through the back propagation algorithm; S6. Take the best model weight tested on the test set and save it, and save the hash code that performs best on the test set.

2. The cross-modal hash retrieval method based on hint learning and adaptive Mamba gated selection fusion according to claim 1 is characterized in that: In step S3, the CLIP framework is used as the feature extraction network in the cross-modal hash retrieval architecture based on prompt learning and adaptive Mamba gating selection. The image feature extraction module adopts the Vision Transformer network, and the text feature extraction module selects the GPT2 network with ViT-B-16 pre-trained weights.

3. The cross-modal hash retrieval method based on hint learning and adaptive Mamba gated selection fusion according to claim 1 is characterized in that: In step S3: in the cross-modal hash retrieval architecture based on hint learning and adaptive Mamba gated selection fusion, a two-branch Transformer is used to extract image and text features; a Transformer network for extracting hint features is used to extract hint weight similarity.

4. The cross-modal hash retrieval method based on hint learning and adaptive Mamba gated selection fusion according to claim 1 is characterized in that: In step S3: In the cross-modal hash retrieval architecture based on hint learning and adaptive Mamba gated selection, the pre-trained ViT is used as the visual encoder to extract the image-text representation Use pre-trained GPT2 as a text encoder to extract image and text representation The constructed mask prompt sentence tokens are embedded into the feature representation as A two-layer Transformer is used to construct the prompt learning module, in which the prompt encoder is responsible for adapting and fusing the prompt features with the original text features.

5. The cross-modal hash retrieval method based on hint learning and adaptive Mamba gated selection fusion according to claim 1 is characterized in that: In step S3: the cross-modal hash retrieval architecture based on hint learning and adaptive Mamba gated selection includes a text affinity guided hint module, which can dynamically learn and distinguish foreground information that is helpful for retrieval, and optimize text feature representation through a dynamic hint mechanism; There is a category label set l i Corresponding to the image-text pair i and t ,The generation of prompt text includes the following steps: ,The prompt generation is guided by the initial context template, which is in the form of: where {class_name i } represents label l i The category name is obtained by combining the context template with the category tag name class_name i Combine to generate a complete prompt word: prompt i ="This is an image containing{class_name i }" After constructing the prompt word, use the tokenizer to tokenize the prompt word prompt i Perform BPE encoding to obtain the token sequence: token_prompt i =Tokenizer(prompt i ) By encoding the token sequence, we get the embedding vector p of the prompt word i : where d p is the embedding dimension of CLIP; Finally, the resulting cue embedding is combined with the learnable context vector C ctx Combined to form the final prompt text features: f i p =p i +C ctx Among them C ctx It is a learnable context vector. By dynamically generating a learnable Mask mask corresponding to the prompt word, it can help the network dynamically learn to distinguish the feature information in the image and text that is beneficial to the retrieval results, dynamically distinguish the foreground and background information in the image, and use the dynamic Mask mechanism to dynamically cover the background information in the text, retain the foreground features, and at the same time, the learned dynamic prompts are weighted with the original text to reduce the weight of the features in the original text that are irrelevant to the retrieval results and increase the weight ratio of the features that are beneficial to the retrieval; the generated prompt text f i p As the input of the TAP module; the TAP module is mainly composed of two layers of Transformer encoders, which are responsible for fusing the prompt text and the original text features; through the Transformer layer in the TAP module, the prompt text features f i p and the original text features f i t , which is expressed as: The fused prompt text feature matrix and the original text feature matrix Perform element-by-element multiplication and obtain weighted feature representation through global average pooling operation: Among them, AvgPool represents the global average pooling, f i pt is the final weighted text feature representation.

6. The cross-modal hash retrieval method based on hint learning and adaptive Mamba gated selection fusion according to claim 5 is characterized in that: In step S3: first, the image features and the obtained mixed prompt text features are filtered and the cross-modal features are fused to obtain the fused features, and the following formula can be obtained: f i fusion =Concat(SiLU(Mlp(LN(f i v ))),SiLU(Mlp(LN(f i t )))) Among them, Concat represents concatenation operation, LN represents regularization operation, Mlp represents multi-layer perceptron, and SiLU represents activation function; Then the fused features are sent to the Mamba adaptive selection module, namely: Among them, Flipa xis＝i ,i∈{1,2} indicates flipping along a specific axis. T represents the reverse operation of flipping, SSM represents the state space model; Then, the three features obtained are subjected to adaptive weighted filtering as follows: Among them, temp is the learnable temperature coefficient, which is used to balance the weights of the four features before and after flipping, and θ represents the learnable parameter; Finally, the filtered features are passed through a layer of Transformer encoder to obtain: The obtained features It represents the features obtained by adaptive Mamba gating selection fusion module, and Split represents the separation of individual image and text features from the fusion features.

Images