Efficient cross-modal hashing retrieval framework and optimization method based on CLIP feature fusion

Abstract

The application discloses an improved cross-modal hash retrieval framework (IFFH) which is composed of a carefully designed double-branch text-image fusion network ITJN as shown in the following figure 2 and a multi-semantics alignment hash loss function. The framework adopts a CLIP architecture for feature extraction, enhances local detail capture through a fusion strategy of residual factorized attention and convolution operation, and realizes multi-semantics alignment in combination with a multi-label proxy loss, a modality-independent pair loss and a second-order spherical mutual information loss function. The framework has the advantages of high feature fusion efficiency, accurate semantics alignment and excellent retrieval performance, and the retrieval performance on MIRFLICKR-25K and NUS-WIDE datasets is improved by 4-6% compared with existing methods.

Description

Technical Field

[0001] This invention belongs to the field of deep cross-modal hash retrieval, specifically relating to a feature fusion-based cross-modal hash retrieval method that enables fast and accurate cross-modal retrieval from massive datasets. [1] Background Technology

[0002] In recent years, with the rapid development of internet technology and mobile devices, massive amounts of multimedia data have exploded. Users generate and upload large amounts of heterogeneous data such as images, videos, audio, and text every day. How to efficiently store and quickly retrieve the required information has become a critical problem that urgently needs to be solved. [2] Hash-based nearest neighbor search technology has gradually become a research hotspot due to its low storage cost and high retrieval efficiency. However, current mainstream deep cross-modal hashing methods suffer from the following shortcomings: Imbalance in cross-modal matching: Due to the significant differences in feature representation between image and text modalities, existing methods struggle to achieve accurate alignment. The amount of image data during training is typically much larger than that of text data, leading to model bias towards the image modality. Limitations in retrieval accuracy: Existing methods mainly rely on local semantic optimization, making it difficult to guarantee global semantic consistency. [3] Furthermore, existing methods struggle to capture complex semantic relationships between samples when dealing with multi-label scenarios, resulting in semantically similar samples being far apart in the hash space.

[0003] Existing solutions include: Jiang et al. [4] In 2017, Li et al. first proposed Deep Cross-Modal Hashing (DCMH), which combines feature learning and hash encoding into a single framework. It directly learns image and text hash codes through discrete optimization while maintaining consistency in semantic feature space and Hamming space similarity. [5] A self-supervised adversarial hashing (SSAH) method was proposed, employing two adversarial networks to maximize semantic relevance. Its core is based on the semantic relevance of image and text modalities reserved by a label network. In 2022, Wang et al. [6] Fast Cross-Modal Hashing (FCMH) was proposed to explore the similarity between global and local information in data, and a discrete hashing learning scheme was put forward. In 2024, Qin et al. [7]We propose to introduce the Quadratic Spherical Mutual Information Loss (QSMI) function to alleviate the semantic gap in cross-modal retrieval, and use the squared truncation method to optimize the model training process. At the same time, we use a multi-head self-attention encoder to extract informative semantic representations.

[0004] However, these methods still suffer from inaccurate semantic alignment and insufficient feature extraction when dealing with complex multi-label data, especially performing poorly when handling large-scale real-world data. Therefore, there is an urgent need for a novel cross-modal hashing retrieval method that can effectively handle modality imbalance and improve feature fusion performance. This is why the CLIP framework is needed. [8] As an information extraction framework, through Vision Transformer [9] and BERT encoder

[10] Feature representations of images and text are extracted separately. A self-designed ITJN fusion strategy is adopted, along with a designed multi-semantic alignment network, to achieve a deeper understanding of semantic information, resolve the imbalance between image and text data, and ultimately improve retrieval accuracy.

[0005] [1]Chen W, Liu Y, Wang W, et al. Deep Learning for InstanceRetrieval: A survey[J]. IEEE Transactions on Pattern Analysis and MachineIntelligence, 2022, 45(6): 7270-7292. [2] Wang Yongxin. Research on hash methods for large-scale cross-modal retrieval [D]. Shandong University, 2021. [2]Huo YD, et al. Deep Semantic-Aware Proxy Hashing for Multi-labelCross-Modal Retrieval. IEEE Transactions on Circuits and Systems for VideoTechnology, 2023, 34(1): 576-589. [[4]Jiang QY, Li W J. Deep Cross-Modal Hashing[C]. IEEE Conference on Computer Vision&Pattern Recognition. IEEE Computer Society, 2017:3270-3278. [5]Li C, Deng C, Li N, et al. Self-Supervised Adversarial HashingNetworks for Cross-Modal Retrieval[J]. Proceedings of The IEEE Conference onComputer Vision and Pattern Recognition, 2018. [6]Wang Y, Chen Z D, Luo X, et al. Fast Cross-Modal Hashing WithGlobal and Local Similarity Embedding[J]. IEEE Transactions on Cybernetics,2021, 99:1-14. [7]Qin Q B, et al. Deep Neighborhood-preserving Hashing withQuadratic Spherical Mutual Information for Cross-Modal Retrieval. IEEETransactions on Multimedia, 2024. [8]Y. Zhuo, Y. Li, J. Hsiao, C. Ho, and B. Li, “CLIP4Hashing:Unsupervised Deep Hashing for Cross-Modal Video-Text Retrieval,” in Proc.Int. Conf. Multimedia Retr., Jun. 2022, pp. 158–166. [9]Dosovitskiy A, Beyer L, Kolesnikov A, et al. An Image is Worth16x16 Words: Transformers for Image Recognition at Scale[C]. InternationalConference on Learning Representations, 2021.

[10] Devlin J, Chang MW, Lee K, et al. BERT: Pre-training of DeepBidirectional Transformers for Language Understanding[C] / / Conference of theNorth American Chapter of the Association for Computational Linguistics:Human Language Technologies, 2-7 June 2019, Minneapolis, Minnesota, USA.2019. Technical issues

[0006] Existing cross-modal hash retrieval methods suffer from three main technical problems: First, existing CLIP-based methods rely excessively on CLIP's natural alignment capabilities, neglecting deep semantic connections between modalities, resulting in hash codes failing to effectively maintain the similarity structure of the original high-dimensional data space. Second, traditional methods using pairwise loss or triplet loss can only process a single pair of positive and negative samples at a time, only locally capturing data similarity, requiring multiple iterations to learn a suitable hash function, leading to low learning efficiency. Finally, existing frameworks perform poorly in terms of model convergence speed and training time, incurring high computational costs and affecting practical application performance. Technical solution

[0007] The purpose of this invention is to provide a cross-modal hash retrieval method based on feature fusion to solve the problems of unbalanced cross-modal matching and limited retrieval accuracy in the prior art.

[0008] To achieve the above objectives, this invention provides the following technical solution: a cross-modal hash retrieval method based on feature fusion, comprising a feature extraction module, a feature fusion module, and a loss optimization module. The feature extraction module adopts a CLIP architecture, extracting feature representations of images and text using a Vision Transformer and a BERT encoder, respectively. Preferably, the feature fusion module employs a dual-path structure: the first path performs multi-scale feature fusion using a multi-layer perceptron, and the second path combines a multi-head self-attention mechanism and a feedforward network to enhance local detail capture. The feature fusion module introduces a factorized attention mechanism and convolution operations, achieving feature enhancement through the formulas FactorAtt(i) and ConvAtt(x), significantly reducing computational complexity. The loss optimization module comprises three parts: multi-label proxy loss, modality-independent pair loss, and quadratic spherical mutual information loss, using a multi-semantic alignment loss function L... MSAOptimization is performed. The multi-semantic alignment loss function ensures semantic consistency of samples in the hash space through multi-label proxies, widens the distance between irrelevant samples through modality-independent pairs, and maximizes the amount of information within and between modes through quadratic spherical mutual information.

[0009] The specific implementation steps are as follows: Step 1: Data preprocessing stage.

[0010] Image data and corresponding text descriptions are divided into three parts: training set, test set, and retrieval set. The images are then preprocessed and uniformly cropped to a size of 224x224 pixels. Step 2: Network architecture construction phase.

[0011] A CLIP architecture is adopted as the backbone network for feature extraction, and a dual-branch feature fusion network, ITJN, is designed. Branch 1: Multi-scale feature fusion is performed through a multi-layer perceptron. Branch 2: Local detail capture is enhanced by combining a multi-head self-attention mechanism and a feedforward network. Factorized attention mechanism and convolutional operations are introduced for feature enhancement.

[0012] Step 3: Loss function optimization stage.

[0013] A Multi-Semantic Alignment Hash (MSAH) loss function is constructed, consisting of three parts: label proxy loss, which ensures that samples maintain semantic consistency in the hash space; modality-independent pair loss, which widens the distance between irrelevant samples; and quadratic spherical mutual information loss, which maximizes the amount of information within and between modes.

[0014] Step 4: Training and Evaluation Phase.

[0015] This invention employs a training set for model training, evaluates model performance on a validation set, saves the optimal model, and tests the model's effectiveness on a test set to generate the final hash code. This approach effectively addresses the cross-modal data imbalance problem through the synergistic effect of a feature fusion network and a multi-semantic alignment loss function, while simultaneously improving retrieval accuracy and training efficiency. The cross-modal hash retrieval method based on feature fusion provided by this invention offers the following advantages:

[0016] 1. This invention effectively solves the problem of cross-modal data imbalance through the synergistic effect of dual-path feature fusion network and multi-semantic alignment loss function, and improves the retrieval performance on MIRFLICKR-25K and NUS-WIDE datasets by 4-6% compared with existing methods.

[0017] 2. This invention adopts a fusion strategy of factorized attention mechanism and convolution operation, which significantly reduces computational complexity and enhances the ability to capture local details, enabling the model to converge to a stable state within 10 training rounds.

[0018] The embodiments of the present invention are described in illustrative terms only, illustrating certain exemplary application scenarios. Undoubtedly, those skilled in the art can modify and optimize the described embodiments without departing from the spirit and scope of the invention. For example, for zero-shot scenarios, transfer learning can be used to enhance the model's generalization ability; for multi-label data, a label relationship matrix can be further designed to improve retrieval performance. Therefore, the above figures and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0019] Fusion strategy optimization: Fine-grained feature alignment between modalities is achieved based on a convolutional attention mechanism. This improves image-text matching accuracy and reduces computational overhead.

[0020] Loss function design: Multimodal irrelevant loss and binary spherical mutual information loss are used to jointly optimize hash code generation, thereby enhancing the retrieval performance of the model. Attached Figure Description

[0021] Figure 1 This figure provides a brief comparison of the model diagram with other model diagrams. It illustrates the structural comparison between the Improved Cross-Modal Hash Retrieval Framework (IFFH) and other models, highlighting the improvements of the dual-branch text-image fusion network and the multi-semantic alignment loss function in feature extraction and semantic alignment.

[0022] Figure 2 This diagram illustrates the implementation details of the feature fusion module in the IFFH framework. It describes how the module employs a multilayer perceptron and factorized convolutional attention mechanism to achieve multi-scale feature fusion and local detail capture, significantly improving semantic alignment capabilities.

[0023] Figure 3 This is a schematic diagram of the model's structure. The diagram illustrates the overall structure of the IFFH framework, including a feature extraction module, a feature fusion module, and a multi-semantic alignment loss optimization module, demonstrating how the framework collaboratively processes cross-modal data.

[0024] Figure 4 This figure shows the NDCG@1000 results for different code lengths on the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets. It illustrates the performance of the IFFH framework with normalized discount cumulative gain (NDCG@1000) at different hash code lengths on the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets, comparing the model's ranking quality in multi-label retrieval. Figure 4The results of IFFH on the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets using NDCG@1000 are shown, which measures the ranking quality of multi-label data in the retrieval results. IFFH performs best on MIRFLICKR-25K, with an NDCG value exceeding 0.85, leading other methods by approximately 6%-8%. Performance decreases on NUS-WIDE and MS COCO, mainly due to the complexity of the label system and the scene. NDCG values ​​for I2T tasks are generally higher than those for T2I tasks, reflecting the advantage of image information in semantic alignment.

[0025] Figure 5 The figure shows that mAP@H ≤ 2 for different code lengths on the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets. This figure illustrates the mean accuracy (mAP) performance on the same dataset for different hash code lengths. Figure 5 Experimental results for mAP@H<2 on three datasets with different hash code lengths are presented. (a)-(c) represent the image-to-text (I2T) task, and (d)-(f) represent the text-to-image (T2I) task. In these two tasks, our method shows particularly significant advantages on MIRFLICKR-25K and NUS-WIDE. At a hash length of 64 bits, our algorithm still outperforms most datasets, but performance declines on the MS COCO dataset. As the hash length increases, the mAP@H<2 value generally decreases, demonstrating that the sparsity of the discrete space affects retrieval accuracy.

[0026] Figure 6 This study investigates the impact of the hyperparameter λ on retrieval performance. By comparing mAP results with different hash coding lengths on different datasets, it can be seen that through dense sampling experiments (λ from 0.1 to 0.9), the model performance reaches its optimal balance point around λ=0.2. In the image-to-text (I2T) task, setting λ=0.2 or λ=0.3 yields the best retrieval performance, especially on the MIRFLICKR-25K dataset. In the text-to-image (T2I) task, the framework performs best across all hash lengths and datasets when λ=0.2. We did try finer-grained hyperparameter tuning (including 0.1, 0.15, 0.2, 0.25, and 0.3), but found no significant performance difference within the 0.15-0.25 range. In conclusion, choosing λ=0.2 as the hyperparameter is reasonable in balancing the overall performance of both I2T and T2I tasks.

Claims

1. A cross-modal hash retrieval method based on feature fusion, characterized in that, It includes a feature extraction module, a feature fusion module, and a loss optimization module; the feature extraction module adopts the CLIP architecture, which extracts feature representations of image and text modalities through Vision Transformer and BERT encoder respectively.

2. The cross-modal hash retrieval method based on feature fusion according to claim 1, characterized in that, The feature fusion module adopts a dual-path structure. The first path uses a multi-layer perceptron for multi-scale feature fusion, combined with a multi-head self-attention mechanism and a feedforward network to enhance the deep understanding of global semantics. The second path uses a factorized convolutional attention mechanism. Convolution operations can enhance the accurate capture of local details, while using factorized attention does not introduce additional computational complexity to the model.

3. The cross-modal hash retrieval method based on feature fusion according to claim 1, characterized in that, The loss function is a multi-semantic alignment loss function. This loss function comprises three parts: L1 multi-label proxy loss, L2 modality-independent pair loss, and L3 quadratic spherical mutual information loss. Through the multi-semantic alignment loss function L... MSA = λ1L1 + λ2L2 + λ3L3 for optimization.

Images