CLIP-based deep joint semantic alignment unsupervised image-text retrieval method

Abstract

The invention discloses an unsupervised image-text retrieval method for deep joint semantic alignment based on CLIP, and relates to the technical field of image-text retrieval methods, comprising the following steps: S1, CJSAH extracts features of images and texts from a pre-trained CLIP backbone network; s2, respectively calculating feature similarity matrixes in the image and text modals in batches; s3, the feature similarity matrix is enhanced and fused into a joint modal similarity matrix which is used for supervising learning of hash codes; s4, after the features are spliced, semantic interaction is carried out through a Transf ormer encoder, and then comparison loss is calculated by disassembling the features subjected to modal fusion; when the comparison loss is calculated in the step S4, the CJSAH further introduces a momentum comparison learning module which comprises a momentum encoder and a dynamic queue so as to mine negative sample information in a larger range; experiments carried out on three widely used data sets show that the CJSAH provided by the invention obtains a satisfactory result in the aspect of total retrieval accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of graphic and text retrieval methods, and particularly to an unsupervised graphic and text retrieval method based on deep joint semantic alignment of CLIP. Background Art

[0002] With the rapid development of the Internet and social media, multi-modal data has increased exponentially. Traditional single-modal retrieval methods are difficult to meet users' access requirements for various modal data. This explosive growth of data not only poses huge challenges to data management, but also provides new opportunities and challenges for data retrieval and cross-modal retrieval.

[0003] Data retrieval is the bridge connecting data management and cross-modal retrieval. Traditional data retrieval methods mainly target single-modal data, such as text retrieval or image retrieval. However, with the explosion of multi-modal data, the limitations of single-modal retrieval have become increasingly prominent. This has prompted researchers to start exploring cross-modal retrieval technologies to meet users' comprehensive retrieval requirements for different modal data.

[0004] As an advanced form of data retrieval, cross-modal retrieval not only inherits the basic principles of traditional retrieval, but also breaks through the boundaries of modalities. It can achieve interaction between modalities, promote semantic matching, and make full use of the complementarity and consistency of different modal data. For example, users can retrieve relevant images by inputting text, or query relevant video content by images. This cross-modal retrieval ability greatly enriches users' retrieval experience and improves the efficiency of information acquisition.

[0005] In the context of big data, cross-modal retrieval faces huge challenges, especially in terms of processing efficiency and storage cost. To solve this problem, cross-modal retrieval methods based on hashing have emerged. This method maps multi-modal media data to a common Hamming space, significantly improving the retrieval speed and storage efficiency. Semantically similar instances of different modalities can be represented by similar hash codes, and their Hamming distance effectively measures the correlation between instances.

[0006] The hashing method utilizes data compression and indexing techniques in big data management to transform high-dimensional multimodal data into low-dimensional binary representations. It provides an efficient implementation for cross-modal retrieval, making it possible to perform fast retrieval in a vast amount of multimodal data. Cross-modal hashing retrieval realizes the interaction between modalities, promotes semantic matching, and fully exploits the complementarity and consistency of different modality data. The hashing-based cross-modal retrieval method significantly improves the retrieval speed and storage efficiency by mapping multimodal media data to a common Hamming space. Semantically similar instances from different modalities can be represented by similar hash codes, and their Hamming distance effectively measures the correlation between instances. Therefore, binary vectors have become a research hotspot in terms of storage and mutual Hamming distance calculation.

[0007] Most traditional hashing methods mainly focus on single-modal retrieval, where the query and the database are homogeneous. For example, both the query item and the database item are images. However, in real-world applications, data usually contains multiple types, such as text, images, audio, or video, so it becomes necessary to perform retrieval across different modalities.

[0008] In cross-modal hashing retrieval, the main goal is to learn a common binary code representation for data samples from different modalities. This task faces two main challenges: (1) How to effectively alleviate the semantic gap between heterogeneous modalities during the hash code learning process; different modalities may have different levels of abstraction and semantics. For example, images may contain specific visual features, while text descriptions may contain more abstract concepts. The cross-modal hashing model needs to determine a common space to bridge the semantic gap and learn a general representation that can capture the joint semantics between different modalities. (2) How to preserve the correlation of heterogeneous data in the binary code. The hash codes learned by the cross-modal hashing model need to have the ability to distinguish data correlations. For example, an image and its corresponding text description with similar semantic labels should have similar binary codes.

[0009] To address the above challenges, a variety of techniques have been proposed. According to whether labels are used, these methods can be divided into two categories: supervised and unsupervised. Supervised methods utilize pre-annotated labels to preserve the correlation between samples from different modalities, thereby learning more discriminative representations. These methods directly use the pre-annotated labels as a unified guidance to learn the required hash codes. However, annotating large-scale datasets is a time-consuming and arduous task limited by the annotation quality. In contrast, unsupervised methods do not rely on manual annotation but generate hash codes by mining the relevant structures in the data itself. Although unsupervised methods have made some progress in the absence of high-quality supervision signals, their effects are relatively weak. However, unsupervised methods have more promising prospects for real-world applications because it is difficult to obtain manual annotations.

[0010] In the past few years, many unsupervised hashing methods have been developed to achieve efficient cross-modal retrieval. These methods project the features of heterogeneous modalities into a shared Hamming space based on a series of learning strategies such as matrix factorization, graph regularization, and deep learning, while preserving semantic relevance. However, there is a problem of weak semantic interaction between different modalities in previous studies, which may lead to the learned hash codes not being able to well preserve the correlation of heterogeneous data. In addition, due to the small scale of the dataset, these methods are highly dependent on the training data, resulting in limited learned features and raising doubts about the effectiveness of the model in actual deployment.

[0011] Recently, Contrastive Language-Image Pretraining (CLIP) has achieved great success. This model has learned a powerful semantic understanding ability to connect images and texts through pre-training on a large-scale image-text pair dataset. The success of CLIP has promoted the development of the cross-modal field and demonstrated good zero-shot and few-shot effects on multiple downstream tasks. For example, SAM-CLIP in the field of semantic segmentation, Multi-CLIP applied to 3D scene question answering, and RemoteCLIP applied to the remote sensing multi-modal field.

[0012] However, in the field of cross-modal hashing retrieval, the potential of CLIP has not been fully explored. Summary of the Invention

[0013] The purpose of the present invention is to provide an unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP, using the visual and language features extracted by CLIP. To further narrow the modality gap between image and text features, we design a simple and effective cross-modal feature interaction module. To better capture the cross-modal semantic association between instances, we construct a joint modality similarity matrix. In addition, we integrate Momentum Contrastive Learning (MCL) to improve the representation ability by introducing a dynamic queue to expand the scale of negative sample pairs. Experiments conducted on three widely used datasets show that the proposed CJSAH has achieved satisfactory results in terms of total retrieval accuracy.

[0014] To solve the above technical problems, an unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP provided by the present invention includes the following steps:

[0015] S1. CJSAH extracts the features of images and texts from a pre-trained CLIP backbone network;

[0016] S2. Calculate the feature similarity matrices within the image and text modalities respectively within a batch;

[0017] S3. Enhance and fuse the feature similarity matrices into a joint modality similarity matrix for supervising the learning of hash codes;

[0018] S4, after the features are concatenated, semantic interaction is performed through the Transformer encoder, and then the contrast loss is calculated by disassembling the features after modal fusion;

[0019] When calculating the contrast loss in S4, CJSAH also introduced a momentum contrastive learning module, which includes a momentum encoder and a dynamic queue to mine negative sample information on a larger scale.

[0020] Furthermore, the method for constructing the joint modality similarity matrix in S3 is:

[0021] Construct high-order semantic associations between preserved instances and filter out noise features. Suppose a batch of input O′={(x i ,y i |i∈[1,n])}, first create a primary joint similarity matrix:

[0022] S′=A1S I +(E-A1)S T

[0023] Among them, S I ∈[-1,+1] n×n and S T ∈[-1,+1] n×n Represents the similarity matrix related to image and text modalities, A1∈[-1,+1] n×n , is the matrix balancing factor, is a hyperparameter, represents the identity matrix;

[0024] For the text modality, the features extracted by the pre-trained CLIP model are used to construct the cosine similarity matrix S T , where each element is calculated as follows:

[0025] S Tij =cos(X Ti* ,X Tj* )

[0026] Here X Ti* and X Tj* Represents the i-th and j-th features of the text modality, where cos(X Ti* ,X Tj* ) represents the cosine similarity of the i,jth feature of the text modality;

[0027]

[0028] Similarly, for the image modality, we use the image features extracted by the pre-trained CLIP model to construct the Euclidean distance similarity matrix S I, where the calculation method of each element is as follows:

[0029]

[0030] Here, ρ is a scaling factor used to adjust the quantization region;

[0031] Optimize the primary joint similarity matrix through the sigmoid function:

[0032]

[0033] Here represents a matrix of all 1s, while represents the identity matrix.

[0034] 4. Further, the multi-modal feature fusion in S3 is specifically as follows:

[0035] Perform multi-modal feature semantic interaction through the Transformer Encoder module;

[0036] First, concatenate the features extracted by the pre-trained CLIP model to obtain the multi-modal feature

[0037] d = d v + d t ;

[0038] Then, use f as the input of the Transformer module and utilize the self-attention mechanism to capture the feature semantic associations within and between modalities;

[0039] In the self-attention mechanism, use the multi-modal feature f to construct queries, keys, and values, which are represented by the following mathematical formulas:

[0040] f v = CLIP visual (x i ),

[0041] f t = CLIP textual (y i ),

[0042] f = concat[f v , f t ,

[0043] q i = f i W Q , k j = f j W K , vj = f j W V ,

[0044] Here the transformation matrices in the separate training, d k and d v are the dimensions of keys and values respectively. For all input image - text pair multimodal features, the multimodal Transformer Encoder can generate corresponding features f':

[0045]

[0046] m represents the batch size, SAtt(q i , K, V) represents the self - attention operation, where K and V represent the keys matrix and values matrix respectively, and the feed - forward neural network and residual connection are as follows:

[0047]

[0048] RS1(.) = Norm(Drop(·)+f i ′ ),

[0049] RS2(·) = Norm(Drop(FFN(·))+(·)),

[0050] where Norm(.) represents layer normalization and Drop(.) represents the Dropout function, represents the output of the j - th layer of the multimodal Transformer Encoder, represents the finally obtained multimodal semantic alignment feature.

[0051] Furthermore, in the momentum contrastive learning module in S4, two dynamic queues Q t and Q v are introduced, which are used to store negative example samples of text and images respectively; at the same time, two key - value momentum encoder parameters are created, which are and

[0052] For the data in the text modality, we use the dynamic queue Q t , where k represents the length of the dynamic queue. During the training process, the query image feature of the current batch and the key - value text feature of the corresponding text after passing through the momentum encoder (i.e., positive sample data) calculate the dot product At the same time, the dynamic queue Q tCalculate the dot product between the stored negative samples and the query image features Subsequently, the key value features of the current batch are enqueued, and at the same time, the data of the oldest batch is dequeued.

[0053] Furthermore, the update speed of the momentum encoder is slow and consistent. A strategy for updating the parameters of the momentum encoder is adopted, and the specific expression is as follows:

[0054]

[0055] Among them, m ∈ [0, 1) is the momentum coefficient, and m = 0.99 is set;

[0056] The InfoNCE loss is adopted, and the optimization goal is to align the semantic features of images and texts, so that the features of paired images and texts are as close as possible, while the features learned by unpaired image-text pairs are as far away as possible. The contrast loss function for the fused features is as follows:

[0057]

[0058] m represents the batch size, τ represents the temperature parameter, <.> represents the dot product operation. It should be noted that, and both represent the representations of paired, i.e., jointly semantically aligned, image-text pairs. The goal of the loss function is to maximize the similarity between paired image-text pairs, while making the similarity between unmatched image-text pairs tend to zero. The multi-modal contrast loss can be defined as follows:

[0059]

[0060] Align the multi-modal semantic features of text and image with connected to the hash layer, and the corresponding hash codes of each modality are obtained through the hash layer:

[0061]

[0062] Here, sign(.) represents the sign function, H(.) represents the MLP, and a momentum contrast learning method similar to the features is used. The corresponding loss function is as follows:

[0063]

[0064] The total hash loss is:

[0065]

[0066] Furthermore, in the objective function of hash code learning, the high-order semantic correlation between instances is retained, and the hash code B to be learned is obtained by minimizing the reconstruction loss of the cosine similarity between the joint modality similarity matrix S and B v and B t The hash code is learned by the reconstruction loss of cosine similarity, and the expression of the objective function is as follows:

[0067]

[0068] Furthermore, in order to learn reliable hash codes with stronger discrimination and representation capabilities, the final objective function is the integration of three separate loss functions:

[0069] [[ID=!3]]L = λ1L fusion + λ2L hash + λ3L rec

[0070] where L fusion refers to the multi-modal feature contrast loss, L hash refers to the hash loss, and L rec refers to the instance similarity reconstruction loss, and λ1, λ2, λ3 are balance factors.

[0071] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0072] 1. Based on the features captured by CLIP, we propose an unsupervised deep joint semantic alignment hashing network for cross-modal retrieval. A simple and effective multi-modal fusion Transformer encoder is introduced to obtain rich multi-modal semantics and enhance the interaction between different modalities. A joint modality similarity matrix is constructed to supervise the learning of hash codes. This matrix effectively integrates the original neighborhood relationships from different modalities, enabling the hash codes to highly retain the original semantic correlations between instances.

[0073] 2. In the contrastive learning training, we innovatively introduce momentum contrastive learning (MCL). By introducing a dynamic queue, we expand the scale of negative sample pairs, thereby enhancing the discrimination ability of the learned features for data correlations. Extensive experiments conducted on multiple public benchmark datasets show that the proposed method outperforms unsupervised and supervised SOTA methods in cross-modal hashing retrieval, even when retrieving unseen data. This fully demonstrates the effectiveness of the CJSAH algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a schematic diagram of the basic learning framework of deep joint semantic alignment hashing based on CLIP. DETAILED DESCRIPTION OF THE INVENTION

[0075] Next, we will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0076] Please refer to Figure 1 , the present invention provides an unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP.

[0077] We will briefly review some related cross-modal hashing studies according to whether supervised information (such as semantic labels) is utilized. We will classify the literature into supervised and unsupervised methods.

[0078] Supervised cross-modal hashing

[0079] Generally speaking, supervised cross-modal hashing methods rely on using semantic labels to supervise the learning of hash codes. According to the depth and complexity of the model structure used in the hashing process, these methods can be further divided into shallow or deep cross-modal hashing methods.

[0080] Shallow supervised cross-modal hashing: In the early stage, researchers proposed supervised cross-modal hashing methods to bridge the semantic gap in cross-modal retrieval using shallow learning models. The supervised matrix factorization hashing (SMFH) is a typical method among them. It uses collective matrix factorization to generate unified hash codes, considering both the label consistency of different modalities and the local geometric consistency of each modality.

[0081] Deep supervised cross-modal hashing: Deep neural networks have shown excellent performance in fields such as computer vision, data mining, and information retrieval. For this reason, many deep cross-modal hashing methods have been proposed, and these methods have achieved excellent performance in cross-modal retrieval. The deep cross-modal hashing (DCMH) designs an end-to-end deep learning framework that can simultaneously learn multi-modal features and generate hash codes. The graph convolutional hashing (GCH) is a hash network based on GCN, which uses a novel semantic encoder to maintain semantic consistency during the multi-modal feature encoding process. The deep cross-modal hashing algorithm with joint learning of hash functions and unified hash codes (DCHUC) proposes an end-to-end deep cross-modal learning framework for jointly learning hash codes and functions. The learned hash codes and functions can supervise each other during the optimization process.

[0082] Unsupervised cross-modal hashing

[0083] Unsupervised cross-modal hashing methods aim to learn hash codes and hash functions without the supervision of pre-annotated semantic labels. These methods can also be divided into shallow and deep methods.

[0084] Shallow unsupervised cross-modal hashing: In early unsupervised cross-modal hashing methods, researchers adopted various strategies to solve the problem of learning hash functions without supervision information. Cross-View Hashing (CVH) formulates the problem of learning a hash function as an NP-hard minimization problem and further transforms it into a tractable eigenvalue problem through a novel relaxation method. On the other hand, Collaborative Matrix Factorization Hashing (CMFH) designs unified hash codes using collaborative matrix factorization for cross-modal retrieval.

[0085] Deep unsupervised cross-modal hashing: Deep cross-modal hashing methods have shown excellent performance in cross-modal retrieval tasks without any supervision information and may even achieve results close to supervised hashing methods, making them of great research value. The Deep Binary Reconstruction Hashing Algorithm (DBRC) proposes a deep binary reconstruction network to achieve heterogeneous modal relationship modeling and hash code learning. On the other hand, the Deep Joint Semantic Reconstruction Hashing Algorithm (DJSRH) learns hash codes by reconstructing the joint semantic structure, efficiently integrating the original neighborhood information of multi-modal data. The Joint Distribution Similarity Hashing Algorithm (JDSH) constructs a joint modal similarity matrix and proposes a sampling and weighting scheme to generate more discriminative hash codes. In addition, the Aggregation-based Graph Convolutional Hashing (AGCH) obtains a joint similarity matrix with rich semantics by aggregating structural information and applying different similarity measurement methods.

[0086] Figure 1 CLIP-based Deep Joint Semantic Alignment Hashing (CJSAH) adopts the following basic learning framework: First, CJSAH extracts features of images and texts from a pre-trained CLIP backbone network. Subsequently, the intra-modal feature similarity matrices within the image and text modalities are calculated within a batch and then enhanced and fused into a joint modal similarity matrix for supervised hash code learning. In addition, the features are concatenated and then undergo semantic interaction through a Transformer encoder, and then the contrastive loss is calculated by disassembling the features after modal fusion. When calculating the contrastive loss, CJSAH also introduces a momentum contrastive learning module, which includes a momentum encoder and a dynamic queue, aiming to mine negative sample information in a larger range.

[0087] The proposed method

[0088] The great success of large language models has inspired extensive research on vision - language models (VLMs). These models combine computer vision and natural language processing technologies and can generate a comprehensive understanding of visual and textual information. One of the most prominent models is the Contrastive Language - Image Pretraining (CLIP) model developed by OpenAI. CLIP was pre - trained on approximately 400 million image - text pairs on the Internet, maximizing the consistency between text and images through contrastive learning, thus achieving a profound understanding and processing of text and images.

[0089] Notably, CLIP can learn to represent images and text in a shared embedding space, where the similarity between a given image and text can be measured by their cosine similarity in their respective embedding spaces. One of the main advantages of CLIP is its generality in various downstream tasks without the need for fine - tuning on a specific dataset. This is due to the fact that CLIP is trained on a large number of diverse image - text pairs, covering a wide range of concepts and domains. This inspires us to continuously explore the significance of CLIP for the field of cross - modal hashing.

[0090] Table 1 Main Symbol Definitions

[0091]

[0092] Symbol Definitions

[0093] For the convenience of our description, we first introduce the symbols used in this paper. For convenience, bold uppercase letters (e.g., A) and bold lowercase letters (e.g., a) represent matrices and vectors respectively. Suppose we have a training set: O = {(x i , y i | i ∈ [1, n])}, where represents the image feature of the i - th sample, represents the text feature of the i - th sample, n, d υ and d t represent the number of samples, the dimension of image features, and the dimension of text features respectively. Our ultimate goal is to learn a compact binary representation b i ∈ {-1, 1} 1×k , where k is the length of the hash code. Thus, the hash code matrix can be defined as B = {b i | i ∈ [1, n]}. Table 1 summarizes the above symbols, and the remaining symbols in the table will be introduced in detail later.

[0094] Method Overview

[0095] Figure 1This is the overall model diagram. The Cross-modal Joint Semantic Alignment Hashing (CJSAH) based on CLIP is an innovative learning method. Its core idea is to integrate the features of image and text modalities to achieve higher-level semantic alignment and hash code learning. The design architecture of CJSAH first extracts the features of images and texts through a pre-trained CLIP backbone network. These features are calculated within the batch to generate the feature similarity matrices within the image and text modalities. By enhancing and fusing these similarity matrices, a joint-modal similarity matrix is obtained, laying the foundation for the learning of supervised hash codes.

[0096] Meanwhile, CJSAH also introduces a Transformer encoder for semantic interaction of features. This step makes the semantic relationship between images and texts deeper and more accurate. The features after modal fusion are disassembled to calculate the contrastive loss, further optimizing the learning effect of hash codes. This strategy of feature disassembly helps to better understand the correlation between images and texts, thus improving the semantic expression ability of hash codes.

[0097] During the calculation of the contrastive loss, CJSAH introduces a momentum contrastive learning module, which includes a momentum encoder and a dynamic queue. The purpose of this design is to more comprehensively mine negative sample information and enhance the model's perception of the differences between different semantics. The introduction of the momentum contrastive learning module makes CJSAH more robust when dealing with large-scale data, while effectively improving the stability and convergence speed during the learning process.

[0098] Construction of the Joint-modal Similarity Matrix

[0099] Supervised cross-modal hashing methods accurately describe the relationships between instances through artificial semantic labels. While unsupervised methods generally can only establish similarity metrics between instances based on raw features, which inevitably introduces noisy adjacency relationships, leading to a serious degradation in performance. Therefore, we are committed to constructing high-order semantic associations between instances and filtering out noisy features. Specifically, assuming a batch of input O′={(x i ,y i |i ∈ [1,n])}, we first create a primary joint similarity matrix:

[0100] S′ = A1S I +(E - A1)S T (1)

[0101] where, S I ∈[-1, +1] n×n and S T ∈[-1, +1] n×nrepresent the similarity matrices related to the image and text modalities respectively, where \(A1\in[-1, +1]\). n×n , which is the matrix balance factor and is a hyperparameter. represents the identity matrix

[0102] Previous research [8] has shown that cosine similarity is more suitable for measuring the similarity relationship of text features, while for the image modality, the Euclidean distance similarity method is more appropriate. Therefore, for the text modality, we use the features extracted by the pre-trained CLIP model to construct the cosine similarity matrix \(S\). T , and the calculation method of each element is as follows:

[0103] \(S\) Tij =\(\cos(X\) Ti* , \(X\) Tj* ) (2)

[0104] Here, \(X\) Ti* and \(X\) Tj* represent the \(i\)-th and \(j\)-th features of the text modality respectively. Here, \(\cos(X\) Ti* , \(X\) Tj* ) represents the cosine similarity of the \(i\)-th and \(j\)-th features of the text modality.

[0105]

[0106] Similarly, for the image modality, we use the image features extracted by the pre-trained CLIP model to construct the Euclidean distance similarity matrix \(S\). I , and the calculation method of each element is as follows:

[0107]

[0108] Here, \(\rho\) is the scaling factor used to adjust the quantization region.

[0109] In addition, we optimize the primary joint similarity matrix through the sigmoid function. Due to the characteristics of the sigmoid function, it can further push the instance pairs with semantic similarity closer and pull the instance pairs with different semantics apart, filter out noise features, and enhance the semantic of adjacent correlations.

[0110]

[0111] Here, represents a matrix of all 1s, while represents the identity matrix.

[0112] Multi-modal feature fusion

[0113] We construct a multi-modal semantic fusion module, aiming to align the semantic information of image and text modalities with the joint semantics. We perform multi-modal feature semantic interaction through the Transformer Encoder module

[35] . First, we concatenate the features extracted by the pre-trained CLIP model to obtain multi-modal features d = d v + d t . Then, we use f as the input of the Transformer module and utilize the self-attention mechanism to capture the intra-modal and inter-modal feature semantic correlations. [[ID=1{}]]

[0114] In the self-attention mechanism, we use the multi-modal feature f to construct queries, keys, and values. This can be expressed by the following mathematical formula:

[0115]

[0116] Here, are transformation matrices during separate training, d k and d v are the dimensions of keys and values respectively. For all input image-text pair multi-modal features, the multi-modal Transformer Encoder can generate corresponding features f':

[0117]

[0118] m represents the batch size, and SAtt(q i , K, V) represents the self-attention operation, where K and V represent the keys matrix and values matrix respectively. Next, we have the feed-forward neural network and residual connection, as follows:

[0119]

[0120] Here, Norm(.) represents layer normalization, and Drop(.) represents the Dropout function. represents the output of the j-th layer of the multi-modal Transformer Encoder. represents the finally obtained multi-modal semantic alignment features.

[0121] Momentum contrastive learning

[0122] Most existing contrastive learning training mechanisms rely on setting negative samples based on instances within a batch, and this mechanism is mostly used in the SimCLR learning framework. However, due to the limitation of video memory, this method restricts the scale of negative samples, resulting in the generalization ability of the learned features being restricted. To overcome this challenge, we draw inspiration from the unsupervised visual representation learning method MoCo and innovatively integrate it with the contrastive learning of image-text pair features in CLIP to more extensively mine the information of negative samples, thereby enhancing the feature discrimination level and generalization ability.

[0123] To achieve this integration, we introduce two dynamic queues Q t and Q v , which are used to store negative example samples of text and images respectively. At the same time, we create two key value momentum encoder parameters, namely and The momentum encoder updates slowly and consistently to ensure the consistency of features.

[0124] For data in the text modality, we use the dynamic queue Q t , where k represents the length of the dynamic queue. Since the negative samples stored in the dynamic queue are independent of the batch data and do not need to be stored in the video memory, it can be set very large, greatly enhancing the scale of negative samples. During the training process, the query image features of the current batch are calculated with the corresponding key value text features (i.e., positive sample data) of the momentum encoder to calculate the dot product At the same time, the negative samples stored in the dynamic queue Q t are calculated with the query image features to calculate the dot product Subsequently, the key value features of the current batch perform the enqueue operation, while the data of the oldest batch performs the dequeue operation. The processing method for the text modality is similar.

[0125] This integration method greatly enhances the scale of negative samples through the application of the dynamic queue, thereby enhancing the performance of contrastive learning and making the learned features more generalizable. At the same time, the slow update of the momentum encoder ensures the consistency of features, providing a more stable training basis for the model.

[0126] To ensure the consistency of features and prevent the model from learning "shortcuts" resulting in training collapse, we adopt a strategy for updating the momentum encoder parameters, and the specific expression is as follows:

[0127]

[0128] Among them, \(m\in[0,1)\) is the momentum coefficient, which usually needs to be set relatively large to maintain the consistency of features, and better results can be obtained in this way. In this paper, we set \(m = 0.99\), which makes the key value features in the dynamic queue, although obtained by the encoder under different parameters, have smooth parameter updates, thus ensuring the continuity of the representation.

[0129] To utilize a larger number of negative samples, we adopt the InfoNCE loss. The optimization objective is to align the semantic features of images and texts, making the features of paired images and texts as close as possible, while making the features of unpaired image-text pairs as far away as possible. The contrast loss function for the fused features is as follows:

[0130]

[0131] \(m\) represents the batch size, \(\tau\) represents the temperature parameter, and \(\langle\cdot\rangle\) represents the dot product operation. It should be noted that, and both represent the representations of paired, i.e., jointly semantically aligned, image-text pairs. The objective of the loss function is to maximize the similarity between paired image-text pairs, while making the similarity between unmatched image-text pairs tend to zero. In this way, the model is pushed to learn more semantically aligned multimodal representations, thereby enhancing the generalization ability and the consistency of the representation. Therefore, the multimodal contrast loss can be defined as follows:

[0132]

[0133] We align the multimodal semantic features of text and image with connected to the hash layer, and through the hash layer, the corresponding hash codes of each modality are obtained:

[0134]

[0135] Here, \(\text{sign}(.)\) represents the sign function, and \(H(.)\) represents the MLP. Similar to the features, the momentum contrast learning method is used, and the corresponding loss function is as follows:

[0136]

[0137] The total hash loss is:

[0138]

[0139] Learning hash codes supervised by the joint modality similarity matrix

[0140] In the objective function of hash code learning, we draw on the experience of using similarity matrices in previous studies, including methods such as DJSRH, JDSH, AGCH, DAEH, and CIRH. We retain the high-order semantic correlation between instances, which provides richer semantic information for our joint-modal similarity matrix. Specifically, our goal is to learn the hash code B by minimizing the reconstruction loss of the cosine similarity between the joint-modal similarity matrix S and the hash code B to be learned v and B t The expression of this objective function is as follows:

[0141]

[0142] The design of this objective function aims to ensure that the learned hash code can maintain the semantic correlation between instances in the embedding space while minimizing the difference from the joint-modal similarity matrix. By continuing the experience of previous methods, we expect to effectively learn a compact and semantically informative hash code representation for images and texts while retaining high-order semantic correlations

[0143] Optimization Algorithm

[0144] To learn a reliable hash code with stronger discrimination and representation ability, our final objective function is the integration of the previous three individual loss functions:

[0145] L = λ1L fusion + λ2L hash + λ3L rec (16)

[0146] where L fusion refers to the multi-modal feature contrast loss, L hash refers to the hash loss, and L rec refers to the instance similarity reconstruction loss, and λ1, λ2, λ3 are balance factors. The overall optimization algorithm is shown in Algorithm 1

[0147]

[0148]

[0149] Experimental Comparison

[0150] To verify the effectiveness of the proposed deep joint semantic alignment hashing (CJSAH) method, we conducted a series of experiments and compared it with several state-of-the-art cross-modal hashing methods on three widely used cross-modal retrieval benchmark datasets (i.e., MIRFlickr, NUS-WIDE, MS COCO).

[0151] Dataset

[0152] MIR Flickr dataset: It contains 25,000 pairs of images and labels, covering 24 different concepts. By excluding pairs with fewer than 20 labels, a total of 20,015 image-label pairs were finally obtained. To maintain consistency with the experimental protocols of other cross-modal hashing methods, we randomly selected 2,000 pairs as the query set, and the remaining 18,015 pairs formed the retrieval set. In addition, we selected 5,000 pairs from the retrieval set as the training set.

[0153] NUS-WIDE dataset: It contains 269,648 images and 81 concepts. To be consistent with the experimental protocols of other cross-modal hashing methods, we selected 186,577 pairs of image-text corresponding to the top 10 most common concepts to construct the experimental dataset. 2,000 pairs of image labels were randomly selected as the query set, and the remaining 184,577 pairs of image-text formed the retrieval set. We also selected 5,000 pairs of image-text pairs from the retrieval set as the training set.

[0154] MS COCO dataset: It consists of 123,287 pairs of image-text, covering 80 independent categories. Similar to MIRFlickr and NUS-WIDE, the dataset partition settings include a training set composed of 5,000 randomly selected instances and a query set composed of 2,000 instances, while the retrieval set consists of a total of 121,287 instances.

[0155] Table 2 summarizes the statistics of these two datasets.

[0156] Table 2 Data statistics of the experimental dataset

[0157]

[0158] Benchmarks and evaluation criteria

[0159] We compared the proposed method with 3 state-of-the-art supervised baselines and 6 state-of-the-art unsupervised baselines:

[0160] Supervised baselines: DLFH, LEMON, BATCH

[0161] Unsupervised baselines: CVH, CMFH, DBRC, DJSRH, AGCH, UCMFH

[0162] We focus on two retrieval tasks: (1) "image-to-text", i.e., searching for relevant texts according to any image query. (2) "text-to-image", retrieving similar images from the text retrieval database. To evaluate the effectiveness of the proposed CJSAH and all comparison methods, we select the widely used mean average precision (mAP) to evaluate the performance of all comparison methods, which can well reflect the ranking information and precision. More specifically, for a set of queries Q = [q1, q2,..., q p , the definition of mAP is as follows:

[0163]

[0164] where p is the number of samples in Q, and the formula for AP i (Average Precision) is:

[0165]

[0166] where P i is the number of samples similar to query q i in the database, n is the number of samples in the database, and P ik is the number of the top-k similar samples retrieved for query q i . φ ik is a binary indicator function, which is φ i = 1 if the k-th retrieved sample is similar to query q ik , otherwise φ ik = 0. If two samples have at least one consistent semantic label, we consider them to be similar.

[0167] Table 3 mAP@50 results of all methods in the MIRFlickr dataset

[0168]

[0169] Table 4 mAP@50 results of all methods in the NUS-WIDE dataset

[0170]

[0171]

[0172] Table 5 mAP@50 results of all methods in the MS COCO dataset

[0173]

[0174] Implementation details

[0175] The Cross-modal Joint Semantic Alignment Hashing (CJSAH) method is implemented by Pytorch on a personal host (CPU: 12th Gen Intel(R) Core(TM) i5-12400F 2.50GHz, GPU: NVIDIA GeForce 4060Ti). The ADAM

[14] optimizer is uniformly adopted, and the learning rate is set to 0.001. During the training process, the learning rate is multiplied by 0.75 every 20 epochs. The number of epochs is set to 300, the batch size is set to 256, the size of the momentum encoder is set to 3072, and the momentum coefficient is set to 0.99.

[0176] Results and Analysis

[0177] To comprehensively evaluate the effectiveness of the proposed method, we compared CJSAH with all baselines. The retrieval results of all baselines are from their original literatures. Tables 3, 4, and 5 show that CJSAH almost exhibits the best performance in all metrics of all datasets, thus fully verifying its effectiveness. CJSAH can better capture and preserve multi-modal semantics and achieve joint semantic alignment, which is the reason for its excellent performance. In particular, CJSAH can outperform many supervised relatively lightweight baselines.

[0178] Summary and Outlook

[0179] In this application, we propose a method of Cross-modal Joint Semantic Alignment Hashing (CJSAH) to solve the large-scale unsupervised cross-modal retrieval task. Our method is experimented on three widely used datasets and achieves satisfactory total retrieval accuracy results. By exploring the significance of CLIP in cross-modal retrieval, we design a simple and effective cross-modal feature interaction module, which well solves the problem of the modality gap. To better capture the cross-modal semantic associations between instances, we continue the previous work and construct a joint modality similarity matrix. In particular, during the training process, we integrate Momentum Contrastive Learning (MCL) and expand the scale of negative sample pairs by introducing a dynamic queue, thereby enhancing the representation ability.

Claims

1. An unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP, characterized in that, It includes the following steps: S1. CJSAH extracts the features of images and texts from a pre-trained CLIP backbone network; S2. Calculate the feature similarity matrices within the image and text modalities respectively within a batch; S3. Enhance and fuse the feature similarity matrices into a joint modality similarity matrix for supervising the learning of hash codes; S4. After the features are concatenated, semantic interaction is performed through a Transformer encoder, and then the contrast loss is calculated by disassembling the features after modality fusion; When calculating the contrast loss in S4, CJSAH also introduces a momentum contrast learning module, which includes a momentum encoder and a dynamic queue to mine negative sample information in a larger range.

2. The unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP according to claim 1, characterized in that: The construction method of the joint modality similarity matrix in S3 is as follows: Construct high-order semantic associations between saved instances and filter out noise features. Suppose a batch of input O′ = {(x i , y i | i ∈ [1, n])} is given. First, create a primary joint similarity matrix: S′ = A1S I +(E - A1)S T Among them, S I ∈[-1, +1] n×n and S T ∈[-1, +1] n×n represent the similarity matrices related to the image and text modalities respectively, A1 ∈[-1, +1] n×n , is the matrix balance factor, which is a hyperparameter, E ∈ R n×n represents the identity matrix; For the features extracted by the pre-trained CLIP model in the text modality, construct the cosine similarity matrix S T , where the calculation method of each element is as follows: Here, and represent the i-th and j-th features of the text modality respectively. Here, represents the cosine similarity of the i-th and j-th features of the text modality; Similarly, for the image modality, we construct the Euclidean distance similarity matrix S using the image features extracted by the pre-trained CLIP model I , where the calculation method for each element is as follows: Here, ρ is a scaling factor used to adjust the quantization region; Optimize the primary joint similarity matrix through the sigmoid function: Here, \(1\in\mathbb{R}\) n×n represents a matrix of all ones, while \(E\in\mathbb{R}\) n×n represents the identity matrix.

3. The unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP according to claim 1, characterized in that: The multi-modal feature fusion in S3 is specifically as follows: Perform multi-modal feature semantic interaction through the Transformer Encoder module; First, the feature f extracted from the pre-trained CLIP model v ∈R dv , f t ∈R dt are concatenated to obtain the multimodal feature f ∈ R d , d = d v + d t ; Then, take f as the input of the Transformer module and use the self-attention mechanism to capture the feature semantic associations within and between modalities; W Q ∈R d×dq , W V ∈R d×dv In the self-attention mechanism, use the multi-modal feature f to construct queries, keys, and values, which are expressed by mathematical formulas as follows: f v = CLIP visual (x i ), f t = CLIP textual (y i ), f = concat[f v , f t , q i = f i W Q , k j = f j W K , v j = f j W V , The W here Q ∈ R d×dq , W V ∈ R d×dv are the transformation matrices in the respective trainings, d k and d v are the dimensions of keys and values respectively. For all input image-text pair multimodal features, the multimodal Transformer Encoder can generate corresponding features f': m represents the batch size, and SAtt(q i , K, V) represents the self-attention operation. K and V represent the keys matrix and the values matrix respectively. The feed-forward neural network and the residual connection are as follows: RS1(.) = Norm(Drop(·) + f i ′), RS2(·)=Norm(Drop(FFN(·))+(·)), Here, Norm(.) represents layer normalization, and Drop(.) represents the Dropout function. represents the output of the j-th layer of the multi-modal Transformer Encoder. represents the finally obtained multi-modal semantic alignment features.

4. The unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP according to claim X, wherein: The momentum contrastive learning module in S4, that is, two dynamic queues Q t and Q v are introduced, which are used to store negative example samples of text and images respectively; at the same time, two key-value momentum encoder parameters are created, which are and For text-modal data, we utilize the dynamic queue Q t , where k represents the length of the dynamic queue. During the training process, the query image features of the current batch compute the dot product with the corresponding key-value text features passed through the momentum encoder (i.e., positive sample data). Meanwhile, the dynamic queue Q t computes the dot product between the stored negative samples and the query image features Subsequently, the key-value features of the current batch are enqueued, while the data of the oldest batch are dequeued.

5. The unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP according to claim 4, characterized in that: The update speed of the momentum encoder is slow and consistent, and a strategy for updating the parameters of the momentum encoder is adopted, and the specific expression is as follows: Among them, m∈[0,1) is the momentum coefficient, and m = 0.99 is set; The InfoNCE loss is adopted, and the optimization goal is to align the semantic features of images and texts, so that the features of paired images and texts are as close as possible, while the features learned by unpaired image-text pairs are as far away as possible. The contrast loss function of the fused features is as follows: m represents the batch size, τ represents the temperature parameter, <.> represents the dot product operation. It should be noted that and both represent the representations of paired i.e., jointly semantically aligned image-text pairs. The objective of the loss function is to maximize the similarity between paired image-text pairs while making the similarity between unmatched image-text pairs tend to zero. The multimodal contrastive loss can be defined as follows: Text and image multimodal semantic alignment features and access the hash layer, and obtain the corresponding hash codes of each modality through the hash layer: Here, sign(.) represents the sign function, H(.) represents the MLP, and a momentum contrast learning method similar to the features is used. The corresponding loss function is as follows: The total hash loss is:

6. The unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP according to claim X, wherein: In the objective function of hash code learning, the high-order semantic correlation between instances is retained, and the hash code B to be learned is obtained by minimizing the reconstruction loss of the cosine similarity between the joint modality similarity matrix S and B v and B t The hash code is learned by the reconstruction loss of the cosine similarity, and the expression of the objective function is as follows:

7. The unsupervised image-text retrieval method based on deep joint semantic alignment of CLIP according to claim 6, characterized in that: In order to learn reliable hash codes with stronger discrimination and representation capabilities, the final objective function is the integration of three individual loss functions: L = λ1L fusion + λ2L hash + λ3L rec where L fusion refers to the multi-modal feature contrast loss, L hash refers to the hash loss, and L rec refers to the instance similarity reconstruction loss, and λ1, λ2, λ3 are balance factors.

Images