Unsupervised contrastive cross-modal hashing retrieval method based on lifelong learning

Abstract

The present application belongs to the technical field of image and text cross-modal processing, and particularly relates to an unsupervised contrast cross-modal hash retrieval method based on lifelong learning, wherein image and text data are respectively encoded through pre-trained DINOv2-Small and BGEv1.5-Small, the encoded features are input into a contrast learning module, an adaptive false negative sample cancellation mechanism based on a hash memory bank is designed to identify potential false negative sample pairs and convert them into positive semantic information, so that the model can correctly pull positive sample pairs and push negative sample pairs, and the discriminative representation learning is enhanced. In addition, in order to avoid the need to retrain the model when new category data arrives, a prompt-based frozen pre-trained model continuous learning strategy is designed, which can dynamically balance the plasticity and stability of the model in the unlabeled stream data scene, effectively alleviating the catastrophic forgetting of hash learning. Considering the dynamic distribution of stream data, the training time and computing resources are saved, and continuous and efficient retrieval of unlabeled stream data is realized.

Description

Technical Field

[0001] This invention belongs to the field of image and text cross-modal processing technology, and particularly relates to an unsupervised contrastive cross-modal hash retrieval method based on lifelong learning. Background Technology

[0002] In recent years, the way information is acquired has been profoundly changing with the popularization of multimedia platforms. The amount of text descriptions, image content, and other multimodal data generated by users on the internet is continuously increasing. This data is not only enormous in quantity but also has complex semantic structures and significant modal differences. Faced with this multi-source and heterogeneous information environment, traditional retrieval methods relying on single-modal content are no longer sufficient to meet practical needs. To achieve more flexible and accurate content-related searches in large-scale scenarios, cross-modal retrieval has gradually become an important research direction in the field of artificial intelligence.

[0003] Among numerous cross-modal retrieval techniques, hash coding is considered an efficient solution for massive datasets due to its advantages of fast query speed and low storage overhead. By mapping data from different modalities to a unified binary space, hash models can utilize Hamming distance to perform fast approximate searches. However, most existing unsupervised hashing methods are still based on the assumption of static data, making it difficult to adapt to the application needs of real-world systems where data is constantly updated and scales up.

[0004] As data generation shifts from offline collection to real-time streaming input, hash models need to be able to continuously absorb new knowledge over time. Traditional batch-based learning methods must repeatedly integrate old data for full training when receiving new data, resulting in significant computational and storage burdens. This paradigm is difficult to implement in resource-constrained or real-time-critical environments. Furthermore, as hash functions are continuously updated, hash codes generated from earlier samples gradually become invalid; without a reasonable incremental optimization mechanism, retrieval performance will decline over time. On the other hand, unsupervised cross-modal hashing often uses contrastive learning to uncover latent semantic relationships, but due to the lack of human labels, the model is prone to introducing false negatives during training, causing semantically similar cross-modal pairs to be misclassified as irrelevant, thus interfering with the construction of the hash space.

[0005] Therefore, this invention aims to provide an unsupervised comparative cross-modal hash retrieval method based on lifelong learning. By combining a continuous learning mechanism with a comparative learning module, it continuously mines cross-modal semantic associations from incremental data without relying on manual annotation, and generates efficient and compact hash codes while avoiding catastrophic forgetting. This enables efficient cross-modal retrieval in scenarios with dynamically changing data distribution. This method can be widely applied in multiple fields to achieve real-time and accurate retrieval of massive heterogeneous data, while significantly reducing storage and computing resource consumption. It provides theoretical support and technical assurance for building adaptive, low-cost cross-modal retrieval systems. Summary of the Invention

[0006] The purpose of this invention is to provide an unsupervised contrastive cross-modal hash retrieval method based on lifelong learning, which can be used to retrieve unsupervised streaming data in the cross-modal retrieval process and solve the false negative sample bias that is common in contrastive learning, while maintaining good retrieval performance and reducing computational resource overhead.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by this invention is as follows: an unsupervised comparative cross-modal hash retrieval method based on lifelong learning, comprising the following steps:

[0008] S1: Sampling k For cross-modal data, including image data I k and text data T k ;

[0009] S2: Use pre-trained image feature extraction network and text feature extraction network to encode the image and text respectively, to obtain real-valued features of the image and real-valued features of the text;

[0010] S3: During the query and retrieval phase, real-valued features are converted into hash codes using a symbolic function;

[0011] S4: Input the real-valued features of the image and text into the contrast learning module. Identify false negative samples through an adaptive false negative sample cancellation mechanism based on a hash memory library, remove them from the negative sample set, add weight coefficients and use them as positive samples to calculate the contrast loss. Use gradient descent to bring similar samples closer and push away dissimilar samples, thereby enhancing the discriminative representation learned by the model.

[0012] S5: In unlabeled streaming data scenarios, the prompt-based frozen pre-trained model continuous learning strategy leverages the powerful generalization ability of pre-trained models to mitigate the catastrophic forgetting problem.

[0013] Preferably, the specific process of encoding the image and text in step S2 to obtain the real-valued features of the image and the real-valued features of the text is as follows:

[0014] S21: Input the raw data into a pre-trained feature extraction network to extract deep features;

[0015] S22: Input the extracted deep features into the learnable multilayer perceptron projection head to map the encoder features into real-valued feature vectors of a specified length.

[0016] Preferably, the specific process of step S3 is as follows:

[0017] S31: The real-valued features of the image are processed by the hash function of the image modality of the hash layer to obtain the discrete hash code;

[0018] S32: Process the real-valued features of the text through the text modal hash function of the hash layer.

[0019] Preferably, the specific process of identifying false negative samples and removing them from the negative sample set based on the adaptive false negative sample removal mechanism using a hash memory library in step S4 is as follows:

[0020] S41: Perform cross-modal feature fusion on real-valued features of images and real-valued features of text;

[0021] S42: Calculate the similarity scores between negative samples in the hash memory and the fused features, aggregate the scores to obtain aggregate scores, and filter out the false negative sample set based on the aggregate scores;

[0022] S43: False negative samples are eliminated from the negative sample set by false negative elimination, and false negative samples are converted into positive sample information by adaptive false negative attraction.

[0023] Preferably, the hash memory library in step S42 is set with a dynamic update mechanism, and the specific formula is as follows:

[0024] ;

[0025] in, It is the momentum factor. It is the index position and anchor point in the hash memory library. The corresponding feature is that ||·||2 is the L2 norm.

[0026] Preferably, the cue-based freezing of the pre-trained model in step S5 effectively avoids catastrophic forgetting by dynamically coordinating the balance between model plasticity and stability. The specific process is as follows:

[0027] S51: Perform basic processing: Divide the input image into several image blocks and embed them into the image blocks. Obtain the image label sequence, and then concatenate a learnable cue vector before the sequence. ,Will The whole thing is used as the input for DINO;

[0028] S52: Insert a set of learnable cue vectors before the text tag sequence. ,Will As input to BGE;

[0029] in, It is the length of the prompt. and These are the dimensions of the image tag sequence and the text tag sequence, respectively;

[0030] S53: Perform incremental processing: Freeze the hint parameters and merge the features extracted by the hash function without hints with those extracted by the hash function with hints.

[0031] Preferably, the image and text hash functions for the incremental stage can be represented as follows:

[0032] ;

[0033] in, It is a feature fusion function. and These are the extracted image and text features. It is a modality-specific multilayer perceptron. and These are the parameters of the multilayer perceptron for the corresponding modality. DINO is the pre-trained DINOv2-small image feature extractor, and BGE is the pre-trained BGEv1.5-small text feature extractor.

[0034] The beneficial effects of this invention include:

[0035] This invention provides a cue-based unsupervised cross-modal hashing spurious negative example elimination method. Addressing the weak adaptability to streaming data and the spurious negative sample pair problem in contrastive learning of traditional unsupervised cross-modal hashing methods, this invention innovatively integrates cue-based continuous learning with an improved contrastive learning mechanism, inventing an unsupervised continuous contrastive hashing learning method to achieve efficient cross-modal retrieval of unlabeled streaming data. Specifically, to address the limitation of traditional models requiring repeated training in incremental data scenarios, a cue-based frozen pre-trained model continuous learning strategy is designed. This leverages the powerful generalization ability of the pre-trained model to mitigate the catastrophic forgetting problem, and soft cue tuning technology is used to quickly adapt to the dynamic distribution of unlabeled streaming data while freezing the backbone network, thus achieving simple and efficient continuous learning. For the spurious negative sample problem, an adaptive spurious negative sample cancellation mechanism based on a hash memory library is designed to identify potential spurious negative samples and transform them into positive semantic information to enhance the model's discriminative representation ability. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the architecture for continuous comparative hash learning provided in an embodiment of the present invention.

[0037] Figure 2 This is a performance degradation result diagram based on natural image data provided in an embodiment of the present invention.

[0038] Figure 3 This is a result graph of the precision-recall line based on the natural graph provided in the embodiment of the present invention. Detailed Implementation

[0039] The following is in conjunction with the appendix Figures 1-3 The present invention will be further described in detail below:

[0040] See appendix Figure 1 As shown, the unsupervised contrastive cross-modal hash retrieval method based on lifelong learning consists of an image feature extraction network, a text feature extraction network, an adaptive false negative sample cancellation mechanism module, a contrastive learning module, and a frozen pre-trained model.

[0041] The image feature extraction network is DINOv2-Small, the text feature extraction network is BGEv1.5-Small, the adaptive spurious negative sample cancellation mechanism module is set to mitigate the spurious negative sample bias by using an adaptive spurious negative sample cancellation mechanism based on a hash memory library, the contrastive learning module is used for hash code learning to freeze the pre-trained model to process unlabeled streaming data based on prompts, and a continuous learning strategy is set.

[0042] Includes the following steps:

[0043] S1: Sampling k For cross-modal data, including image data I k and text data T k ;

[0044] S2: Encode the image and text using the pre-trained image feature extraction network DINOv2-Small and the text feature extraction network BGEv1.5-Small respectively to obtain the real-valued features of the image. and text real-valued features ;

[0045] S3: Using symbolic functions during the query and retrieval phase. Convert real-valued features into hash codes , ,in, n It is the length of the dataset. l It is the hash code length;

[0046] S4: Input the real-valued features of the image and text into the contrast learning module. Identify false negative samples through an adaptive false negative sample cancellation mechanism based on a hash memory library, remove them from the negative sample set, add weight coefficients and use them as positive samples to calculate the contrast loss. Use gradient descent to bring similar samples closer and push away dissimilar samples, thereby enhancing the discriminative representation learned by the model.

[0047] S5: In unlabeled streaming data scenarios, the prompt-based frozen pre-trained model continuous learning strategy leverages the powerful generalization ability of the pre-trained model to alleviate the catastrophic forgetting problem, and uses soft prompting tuning technology to quickly adapt to the dynamic distribution of unlabeled streaming data on the basis of the frozen backbone network, thereby achieving simple and efficient continuous learning.

[0048] In step S2, the image and text are encoded to obtain the real-valued features of the image. and text real-valued features The specific process is as follows:

[0049] S21: Transfer the original image data Input a pre-trained image feature extraction network to extract deep features;

[0050] S22: Transfer the original text data Input a pre-trained text feature extraction network to extract deep features;

[0051] S23: Input the extracted deep features into the learnable multilayer perceptron projection head, mapping the encoder features to length. The real-valued eigenvectors.

[0052] The specific process of step S3 is as follows:

[0053] S31: Hash function for image modalities through hash layer The real-valued features of the image are processed to obtain discrete hash codes. Image modality hash function The specific formula is as follows:

[0054] ;

[0055] in, These are the parameters of an image multilayer perceptron;

[0056] S32: Text Modal Hash Function Through Hash Layer Text modal hash function is used to process real-valued features of text. The specific formula is as follows:

[0057] ;

[0058] in, These are the parameters of a text-based multilayer perceptron.

[0059] The adaptive false negative sample cancellation mechanism based on the hash memory library first processes the real-valued features of the image and the real-valued features of the text. Cross-modal feature fusion is performed, then the similarity scores between negative samples in the hash memory and the fused features are calculated and aggregated. Then, a set of false negative samples is selected based on the aggregated scores. Finally, false negative samples are removed from the set of negative samples through false negative elimination, and false negative samples are converted into positive sample information through adaptive false negative attraction.

[0060] The specific process is as follows:

[0061] ;

[0062] in, It is data augmentation for the corresponding modality. and These are anchor points and Supporting samples.

[0063] Then, real-valued features are extracted using an image and text hashing network and fused across modalities.

[0064] ;

[0065] ;

[0066] in, and Support features for image and text modalities, respectively. It is the first k Cross-modal fusion features of samples This indicates different combinations.

[0067] Subsequently, the definition of the hash memory library is given, and the similarity scores between negative samples and fused features in the memory library are aggregated:

[0068] ;

[0069] ;

[0070] ;

[0071] in M It is a built hash memory library. These are the stored real-valued features. It is the size of the hash memory library. These are aggregation functions, including both mean aggregation and maximum aggregation. It is a sorting function. It is the sorted set of similarity scores.

[0072] according to The spurious negative sample index set can be filtered out by combining TOP-K and threshold strategies.J And by eliminating false negatives, we obtain the negative sample set after removing false negatives. N :

[0073] ;

[0074] ;

[0075] ;

[0076] in Indicates the Top-K candidates. It is the similarity score threshold. k It is the positive sample index corresponding to the anchor point in the hash memory library. This represents the number of negative samples sampled. By combining adaptive false negative attraction and false negative elimination, our contrastive loss can be expressed as:

[0077] ;

[0078] in, It is both an aggregated similarity score and an adaptive weight that transforms FNPs into positive sample information. It is the fusion feature of the anchor point. These are the positive samples corresponding to the anchor points. These are false negative samples that have been converted into positive samples. These are negative samples randomly sampled after filtering. Since the number of false negatives varies depending on the anchor point, a normalization coefficient is introduced. To balance the loss scale at each anchor point, making the training process more stable. By minimizing It can separate false negative samples from the negative sample set and convert them into positive information through adaptive false negative attraction, thereby enabling the model to learn rich and discriminative representations.

[0079] Finally, the dynamic update mechanism of the hash memory library is described as follows:

[0080] ;

[0081] in, It is the momentum factor. It is the index position and anchor point in the hash memory library. The corresponding feature is that ||·||² represents the L2 norm. It's worth noting that in the early stages of training, the random initialization of the hash memory may lead to inaccurate similarity calculations between samples and anchor points, thus affecting the direction of model optimization. To address this issue, a warm-up phase was designed, first using... Optimize the hash function and switch to it only after the hash memory database features have been updated and stabilized. Optimize. It can be represented as:

[0082] ;

[0083] in, It is a similarity function. It's a temperature parameter. It refers to the batch size. This represents the negative sample characteristics of the batch sampling.

[0084] The prompt-based frozen pre-trained model continuous learning strategy effectively avoids catastrophic forgetting by dynamically balancing the model's plasticity and stability. The workflow is as follows:

[0085] Foundation stage: First, the input image is divided into several image patches, and an image label sequence is obtained through image patch embedding. Then, a learnable cue vector is concatenated before this sequence. ,Will The entire sequence serves as input to DINO. Similarly, a set of learnable cue vectors is inserted before the text token sequence. ,Will As input to BGE. It is the length of the prompt. These represent the dimensions of the image tag sequence and the text tag sequence, respectively. The hash functions for the image and text modalities, tuned with soft hints, can be expressed as:

[0086] ;

[0087] in, and It consists of image and text features extracted after piecing together the prompts.

[0088] Incremental Stage: To maintain generalization ability (i.e., resistance to forgetting), the cue parameters are frozen, and the features extracted by the hash function without cue are fused with those extracted by the hash function with cue. The image and text hash functions in the incremental stage can be represented as follows:

[0089] ;

[0090] in, It is a feature fusion function. and These are the extracted image and text features. DINO is a pre-trained DINOv2-small image feature extractor, and BGE is a pre-trained BGEv1.5-small text feature extractor.

[0091] Figure 2 and Figure 3The performance degradation results and precision-recall curves of this invention on two publicly available benchmark cross-modal datasets, MIRFLICKR-25K and NUS-WIDE, are presented respectively. The performance degradation experiment aims to verify whether the lifelong learning module can adapt to the continuously growing environment of unlabeled streaming data and mitigate the catastrophic forgetting caused by the addition of new data categories in the data stream. The MIRFLICKR-25K dataset contains 25,000 image-text pairs, divided into 24 categories. The experiment selected pairs with at least 20 text annotations, resulting in 20,015 sample pairs. The NUS-WIDE dataset contains over 269,000 image-text pairs, divided into 81 categories. The experiment selected 186,557 pairs of instances belonging to the 10 most common categories. Compared with nine methods, namely DJSRH (published by Su et al. at ICCV 2019), JDSH (published by Liu et al. at ACM SIGIR 2020), DSAH (published by Yang et al. at ACM ICMR 2020), DGCPN (published by Yu et al. at AAAI 2021), CIRH (published by Zhu et al. at TKDE 2022), ASSPH (published by Li et al. at ACM MM 2022), UCMFH (published by Xia et al. at Inf. Fusion 2023), UCCH (published by Hu et al. at TPAMI 2023), and SACH (published by Cui et al. at Neural Networks 2024), this invention can adapt to unlabeled streaming data and has better retrieval performance.

[0092] In summary, the unsupervised contrastive cross-modal hash retrieval method based on lifelong learning provided by this invention offers a novel design approach for unsupervised cross-modal retrieval and solves the problems of weak adaptability to unlabeled streaming data and spurious negative sample pair bias in previous methods. False negative samples refer to real positive samples that are mistakenly selected as negative samples in contrastive learning. Image and text data are encoded using pre-trained DINOv2-Small and BGEv1.5-Small models, respectively. The encoded features are input into the contrastive learning module, and an adaptive spurious negative sample cancellation mechanism based on a hash memory library is designed to identify potential spurious negative samples and convert them into positive semantic information. This allows the model to correctly approach similar samples (positive sample pairs) and distance dissimilar samples (negative sample pairs), enhancing discriminative representation learning. Furthermore, to avoid retraining the model when new category data arrives, a cue-based frozen pre-trained model continuous learning strategy is designed. This dynamically balances the model's plasticity and stability in unlabeled streaming data scenarios, effectively mitigating the catastrophic forgetting of hash learning. This invention takes into account the dynamic distribution of streaming data, saving training time and computing resources, and realizing continuous and efficient retrieval of unlabeled streaming data.

Claims

1. An unsupervised comparative cross-modal hash retrieval method based on lifelong learning, characterized in that, Includes the following steps: S1: Sampling k For cross-modal data, including image data I k and text data T k ; S2: Use pre-trained image feature extraction network and text feature extraction network to encode the image and text respectively, to obtain real-valued features of the image and real-valued features of the text; S3: During the query and retrieval phase, real-valued features are converted into hash codes using a symbolic function; S4: Input the real-valued features of the image and text into the contrast learning module. Identify false negative samples through an adaptive false negative sample cancellation mechanism based on a hash memory library, remove them from the negative sample set, add weight coefficients and use them as positive samples to calculate the contrast loss. Use gradient descent to bring similar samples closer and push away dissimilar samples, thereby enhancing the discriminative representation learned by the model. S5: In unlabeled streaming data scenarios, the prompt-based frozen pre-trained model continuous learning strategy leverages the powerful generalization ability of the pre-trained model to alleviate the catastrophic forgetting problem. Step S5, which involves cue-based freezing of the pre-trained model, effectively avoids catastrophic forgetting by dynamically balancing the model's plasticity and stability. The specific process is as follows: S51: Perform basic processing: Divide the input image into several image blocks and embed them into the image blocks. Obtain the image label sequence, and then concatenate a learnable cue vector before the sequence. ,Will The whole thing is used as the input for DINO; S52: Insert a set of learnable cue vectors before the text tag sequence. ,Will As input to BGE; in, It is the length of the prompt. and These are the dimensions of the image tag sequence and the text tag sequence, respectively; S53: Perform incremental processing: Freeze the hint parameters and fuse the features extracted by the hash function without hints and the hash function with hints; The image and text hash functions in the incremental phase can be represented as follows: ； in, and These are the extracted image and text features. It is a modality-specific multilayer perceptron. and These are the parameters of the multilayer perceptron for the corresponding modality. DINO is the pre-trained DINOv2-small image feature extractor, and BGE is the pre-trained BGEv1.5-small text feature extractor.

2. The unsupervised comparative cross-modal hash retrieval method based on lifelong learning according to claim 1, characterized in that, The specific process of encoding the image and text in step S2 to obtain the real-valued features of the image and the real-valued features of the text is as follows: S21: Input the raw data into a pre-trained feature extraction network to extract deep features; S22: Input the extracted deep features into the learnable multilayer perceptron projection head to map the encoder features into real-valued feature vectors of a specified length.

3. The unsupervised comparative cross-modal hash retrieval method based on lifelong learning according to claim 1, characterized in that, The specific process of step S3 is as follows: S31: The real-valued features of the image are processed by the hash function of the image modality of the hash layer to obtain the discrete hash code; S32: Process the real-valued features of the text through the text modal hash function of the hash layer.

4. The unsupervised comparative cross-modal hash retrieval method based on lifelong learning according to claim 1, characterized in that, The specific process of identifying and removing false negative samples from the negative sample set using the adaptive false negative sample removal mechanism based on the hash memory library in step S4 is as follows: S41: Perform cross-modal feature fusion on real-valued features of images and real-valued features of text; S42: Calculate the similarity scores between negative samples in the hash memory and the fused features, aggregate the scores to obtain aggregate scores, and filter out the false negative sample set based on the aggregate scores; S43: False negative samples are eliminated from the negative sample set by false negative elimination, and false negative samples are converted into positive sample information by adaptive false negative attraction.

5. The unsupervised comparative cross-modal hash retrieval method based on lifelong learning according to claim 4, characterized in that, The hash memory library in step S42 is set to have a dynamic update mechanism, and the specific formula is as follows: ； in, It is the momentum factor. It is the index position and anchor point in the hash memory library. The corresponding feature is that ||·||2 is the L2 norm.

Images