Cross-modal hashing model construction and retrieval method based on deep semantic interaction

Abstract

The application relates to the technical field of cross-modal hash retrieval, and provides a cross-modal hash model construction and retrieval method based on deep semantic interaction, which comprises the following steps: obtaining training instances; inputting an image-text pair and a label into a feature extraction module in an initial cross-modal hash model to obtain a first image feature, a first text feature and a label feature; inputting the first image feature, the first text feature and the label feature into a feature interaction module to obtain a second image feature and a second text feature; inputting the first image feature, the first text feature, the second image feature and the second text feature into a hash generation module to obtain an image hash code and a text hash code; obtaining a target loss based on the image hash code and the text hash code, and updating the initial cross-modal hash model to construct a cross-modal hash model. The application improves the accuracy of cross-modal hash retrieval.

Description

Technical Field

[0001] This invention relates to the field of cross-modal hash retrieval technology, specifically to a cross-modal hash model construction and retrieval method based on deep semantic interaction. Background Technology

[0002] In recent years, with the rapid growth of multimodal data, represented by images, videos, audio, and text, on the Internet, the need to quickly retrieve semantically relevant data of different modalities from massive amounts of data has become an urgent requirement. Cross-modal hash retrieval maps data of different modalities from a high-dimensional original feature space to a low-dimensional Hamming space using a hash function, generating corresponding binary representations. Simultaneously, a bitwise XOR operation is performed on the binary data in the Hamming space to calculate the similarity between different modalities. Cross-modal hash retrieval has attracted widespread attention and research due to the small storage space and high computational efficiency of the generated hash codes, becoming one of the most significant emerging technologies in similarity retrieval.

[0003] Traditional cross-modal hashing retrieval methods primarily rely on manually extracted features to generate hash codes. However, the expressive power of manually extracted features is insufficient, leading to hash codes that fail to accurately represent modal information and limiting the accuracy of cross-modal hashing retrieval methods. Supervised cross-modal hashing methods often use class labels as supervision information, defining the learning of the hash function as a classification problem or a loss function optimization problem based on pairwise relationships. However, in practice, images or text are usually associated with multiple semantic labels. In this case, the relationship between images and text becomes more complex, relating not only to the number of shared labels but also to the number of labels that are not shared or are uniquely shared. Therefore, multi-level measurements are needed to describe similarity, which supervised methods often neglect. Deep cross-modal hashing, pairwise guided hashing, and self-supervised adversarial hashing networks also ignore this aspect. Secondly, most existing supervised cross-modal hashing retrieval methods have limited ability to maintain semantic consistency during hash learning and reduce semantic gaps between heterogeneous data, failing to bridge the heterogeneity gap between different modalities. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a method for constructing and retrieving cross-modal hash models based on deep semantic interaction.

[0005] In a first aspect, the present invention provides a method for constructing a cross-modal hash model based on deep semantic interaction, comprising:

[0006] Obtain training instances; the training instances include image-text pairs and multiple labels for the image-text pairs;

[0007] The image-text pair and the label are input into the feature extraction module of the initial cross-modal hashing model to obtain the first image feature, the first text feature, and the label feature;

[0008] The first image feature, the first text feature, and the label feature are input into the feature interaction module in the initial cross-modal hashing model to perform feature interaction and obtain the second image feature and the second text feature.

[0009] The first image feature, the first text feature, the second image feature, and the second text feature are input into the hash generation module in the initial cross-modal hash model to obtain the image hash code and the text hash code;

[0010] The target loss is obtained based on the image hash code and the text hash code. The initial cross-modal hash model is then updated based on the target loss to construct the cross-modal hash model.

[0011] Further, the step of inputting the image-text pair and the label into the feature extraction module of the initial cross-modal hashing model to obtain the first image feature, the first text feature, and the label feature includes:

[0012] The image in the image-text pair is input into a CNN-F network to obtain the first image feature;

[0013] The text in the image-text pair is input into a multilayer perceptron to obtain the first text feature;

[0014] The label is input into a multilayer perceptron to obtain the label features.

[0015] Further, the step of inputting the first image feature, the first text feature, and the label feature into the feature interaction module of the initial cross-modal hashing model for feature interaction to obtain the second image feature and the second text feature includes:

[0016] The first image feature and the label feature are concatenated to obtain the first image label feature;

[0017] The first image label features are input into the transformer encoder to obtain the second image label features;

[0018] The first text feature and the label feature are concatenated to obtain the first text label feature;

[0019] The first text label features are input into the transformer encoder to obtain the second text label features;

[0020] The second image feature and the second text feature are obtained by semantically interacting with the second image label feature and the second text label feature through the semantic interaction module.

[0021] Further, the step of obtaining the second image features and the second text features by performing semantic interaction on the second image label features and the second text label features through the semantic interaction module includes:

[0022] The second image label feature and the second text label feature are concatenated to obtain the image-text concatenation feature;

[0023] The image text concatenation features are input into the transformer encoder to obtain the output of the transformer encoder;

[0024] The output of the transformer encoder is input into the feedforward network to obtain the second image features and the second text features.

[0025] Further, the step of inputting the first image feature, the first text feature, the second image feature, and the second text feature into the hash generation module in the initial cross-modal hash model to obtain the image hash code and the text hash code includes:

[0026] The first image feature, the first text feature, the second image feature, and the second text feature are respectively input into the corresponding hash layers to obtain the first image hash feature, the first text hash feature, the second image hash feature, and the second text hash feature; the hash layer corresponding to the first image feature and the hash layer corresponding to the second image feature share the same weight; the hash layer corresponding to the first text feature and the hash layer corresponding to the second text feature share the same weight.

[0027] The image hash code and the text hash code are generated based on the first image hash feature and the first text hash feature, respectively.

[0028] Furthermore, the hash layer includes a multilayer perceptron and an activation function.

[0029] Further, after obtaining the image hash code and the text hash code, the process includes:

[0030] The similarity between different instances is obtained by using the image hash code and text hash code obtained from different instances.

[0031] Furthermore, obtaining the target loss based on the image hash code and the text hash code includes:

[0032] The target loss is obtained based on the image hash code, the text hash code, the second image feature, the second text feature, the first image hash feature, the second image hash feature, the first text hash feature, and the second text hash feature.

[0033] Further, obtaining the target loss based on the image hash code, the text hash code, the second image feature, the second text feature, the first image hash feature, the second image hash feature, the first text hash feature, and the second text hash feature includes:

[0034] Based on the second image features and the second text features, a cross-modal semantic interaction feature triplet loss is obtained;

[0035] The hash feature triplet loss is obtained based on the second image hash feature and the second text hash feature;

[0036] The feature alignment loss is obtained based on the second image feature, the second text feature, the first image hash feature, the second image hash feature, the first text hash feature, and the second text hash feature;

[0037] The quantization loss is obtained based on the image hash code, the text hash code, the first image hash feature, and the first text hash feature;

[0038] The target loss is obtained based on the cross-modal semantic interaction feature triplet loss, the hash feature triplet loss, the feature alignment loss, and the quantization loss.

[0039] Secondly, the present invention provides a cross-modal hash retrieval method based on deep semantic interaction, comprising:

[0040] The data to be retrieved is input into the cross-modal hash model established by the cross-modal hash model construction method based on deep semantic interaction as described in the first aspect, and the retrieval results are obtained.

[0041] The beneficial effects of the cross-modal hash model construction and retrieval method based on deep semantic interaction described in this invention are: extracting tag features, using the feature interaction module to fully semantically interact image features and text features under the action of tag features, improving the semantic representation of each modality, generating higher quality hash codes, thereby improving the accuracy of cross-modal hash retrieval. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the method for constructing a cross-modal hash model based on deep semantic interaction according to an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the structure of the cross-modal hash model construction method based on deep semantic interaction according to an embodiment of the present invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings.

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] This invention provides a method for constructing a cross-modal hash model based on deep semantic interaction, such as... Figure 1 and Figure 2 As shown, the method includes the following steps:

[0047] 101. Obtain training instances; the training instances include image-text pairs and multiple labels for the image-text pairs.

[0048] Specifically, the training instances are the training data, used o i ={v i ,t i ,l i} represents. Where, o i Let v represent the i-th training instance, n represent the number of training instances, and v i Let t represent the image of the i-th training instance. i The text representing the i-th training instance, l i This represents the label set of the i-th training instance, and also the label set of the i-th image-text pair. In this embodiment of the invention, multiple sets of training instances are included, each set containing both image and text modalities and their corresponding label data. The labels for each image-text pair are in multi-label form. Where, l i =[l i1 ,l i2 ,…,l ic ], l i1 l i2 l ic Let represent the 1st, 2nd, and cth tags of the i-th image-text pair, respectively. If o i If it belongs to the j-th category, then l ij =1, j=[1,2,…,c], otherwise l ij =0.

[0049] 102. Input the image-text pair and the label into the feature extraction module of the initial cross-modal hashing model to obtain the first image feature, the first text feature, and the label feature.

[0050] In one possible implementation, the step of inputting the image-text pair and the label into the feature extraction module of the initial cross-modal hashing model to obtain the first image feature, the first text feature, and the label feature includes:

[0051] The image in the image-text pair is input into a CNN-F network to obtain the first image feature;

[0052] The text in the image-text pair is input into a multilayer perceptron to obtain the first text feature;

[0053] The label is input into a multilayer perceptron to obtain the label features.

[0054] This invention follows the design of Deep Cross-Modal Hashing (DCMH) and Self-Supervised Adversarial Hashing (SSAH) methods, employing a dual-tower architecture as the backbone to encode multimodal data into a latent feature space. Specifically, a pre-trained CNN-F network is used to extract image features. The original CNN-F network consists of five convolutional layers and three fully connected layers. The last fully connected layer of the original CNN-F is replaced with a new fully connected layer with 512 nodes to better adapt to the cross-modal hashing retrieval task, where 512 represents the dimension of the image features extracted by the image feature extraction network. The network parameters of the pre-trained CNN-F are initialized on the ImageNet dataset. The structure of the CNN-F network is shown in Table 1 below.

[0055] Table 1

[0056] Network layer Number of channels nuclear size Step length Number of fills conv1 64 11×11 4 0 conv2 256 5×5 2 2 conv3 256 3×3 1 1 conv4 256 3×3 1 1 conv5 256 3×3 1 1 fc6 4096 fc7 4096 fc8 512

[0057] For the text in the image-text pair, a multilayer perceptron with three fully connected layers is used to perform text modality feature extraction. The last node of the fully connected layer is set to 512, where 512 is the dimension of the text features extracted by the text feature extraction network.

[0058] The structure of the multilayer perceptron for extracting text features is shown in Table 2 below:

[0059] Table 2

[0060] Network layer Number of channels fc1 Length of bag-of-words vectors fc2 4096 fc3 512

[0061] For the labels, a multilayer perceptron with three fully connected layers is used to extract the label features. The last layer of the fully connected layers has 512 nodes, where 512 is the dimension of the label features extracted by the label feature extraction network.

[0062] The structure of the multilayer perceptron for extracting label features is shown in Table 3 below:

[0063] Table 3

[0064] Network layer Number of channels fc1 Number of tags fc2 256 fc3 512

[0065] 103. Input the first image feature, the first text feature, and the label feature into the feature interaction module in the initial cross-modal hash model to perform feature interaction and obtain the second image feature and the second text feature.

[0066] In one possible implementation, the step of inputting the first image feature, the first text feature, and the label feature into the feature interaction module of the initial cross-modal hashing model for feature interaction to obtain the second image feature and the second text feature includes:

[0067] The first image feature and the label feature are concatenated to obtain the first image label feature;

[0068] The first image label features are input into the transformer encoder to obtain the second image label features;

[0069] The first text feature and the label feature are concatenated to obtain the first text label feature;

[0070] The first text label features are input into the transformer encoder to obtain the second text label features;

[0071] The second image feature and the second text feature are obtained by semantically interacting with the second image label feature and the second text label feature through the semantic interaction module.

[0072] In one possible implementation, obtaining the second image features and the second text features by performing semantic interaction on the second image label features and the second text label features through the semantic interaction module includes:

[0073] The second image label feature and the second text label feature are concatenated to obtain the image-text concatenation feature;

[0074] The image text concatenation features are input into the transformer encoder to obtain the output of the transformer encoder;

[0075] The output of the transformer encoder is input into the feedforward network to obtain the second image features and the second text features.

[0076] Specifically, the first image feature f I and label features f LThe images are stitched together to obtain the first image label features f. IL The first image label features are input into the transformer encoder, which uses a self-attention mechanism to capture the semantic correlation between image features and label features, and fully utilizes label information to obtain the second image label features f. i v ,as follows:

[0077]

[0078] In the formula, m represents the batch size, SAtt represents the self-attention mechanism, and K... v and V v These are the key matrix and the value matrix, respectively. f represents the query vector for the attention mechanism. i IL This represents the first image label feature corresponding to the i-th training instance. For trainable parameters, R represents real numbers with superscript d×d k d represents the size of the matrix. k The dimension representing the key in the self-attention mechanism. yes transpose, It is a vector. This represents the features obtained by concatenating image features and label features. For trainable parameters, It is the j-th vector in the value matrix. For trainable parameters, d×d v d represents the size of the matrix. v The dimension representing the value of the self-attention mechanism.

[0079] The text modality processing method is the same as that for image modality. First, the first text feature f is processed... T With label features f L By concatenating the features, the first text label feature f can be obtained. TL The first text label feature is input into the transformer encoder, which uses a self-attention mechanism to capture the semantic correlation between text features and label features, and fully utilizes label information to obtain the second text label feature f. i t ,as follows:

[0080]

[0081] In the formula, m represents the batch size, SAtt represents the self-attention mechanism, and K... t and V tThese are the key matrix and the value matrix, respectively. f represents the query vector of the self-attention mechanism. i TL This represents the first text label feature corresponding to the i-th training instance. yes transpose, This represents the features obtained by concatenating text features and label features. Let j be the j-th vector of the value matrix, where, These are trainable parameters.

[0082] In the semantic interaction module, semantic interaction is performed between the second image label features and the second text label features. Specifically, a transformer encoder and a feedforward network are used to further enable full interaction between the image features and the text features, as follows:

[0083] f = concat(f) v ,f t )

[0084]

[0085] q i =f i W Q

[0086] k j =f j W K

[0087] v j =f j W V

[0088] z i =RS2(RS1(f′) i ))

[0089] RS1(.)=Norm(Drop(.)+f′ i )

[0090] RS2(.)=Norm(Drop(FFN(.))+(.))

[0091] In the formula, These are trainable parameters. f represents the output of the semantic interaction module. v f represents the second image label feature. t f′ represents the second text label feature. i f i and fj Features extracted via the self-attention mechanism, q i k represents the query vector for the self-attention mechanism. j The key vector representing the self-attention mechanism. Indicates k j transpose, v j f represents the value vector of the self-attention mechanism. i and f j This represents the feature vector after concatenating the second image label features and the second text label features. RS1 represents the random deactivation function and the layer normalization function. RS2 represents the feedforward neural network, the random deactivation function, and the layer normalization function. Drop(.) represents the random deactivation function. FFN(.) represents the feedforward neural network. (.) represents a part of the corresponding formula. The (.) in each formula corresponds to the previous and next parts. Norm() represents the layer normalization.

[0092] 104. Input the first image feature, the first text feature, the second image feature, and the second text feature into the hash generation module in the initial cross-modal hash model to obtain the image hash code and the text hash code.

[0093] In one possible implementation, the step of inputting the first image feature, the first text feature, the second image feature, and the second text feature into the hash generation module in the initial cross-modal hash model to obtain the image hash code and the text hash code includes:

[0094] The first image feature, the first text feature, the second image feature, and the second text feature are respectively input into the corresponding hash layers to obtain the first image hash feature, the first text hash feature, the second image hash feature, and the second text hash feature; the hash layer corresponding to the first image feature and the hash layer corresponding to the second image feature share weights; the hash layer corresponding to the first text feature and the hash layer corresponding to the second text feature share weights.

[0095] The image hash code and the text hash code are generated based on the first image hash feature and the first text hash feature, respectively.

[0096] In one possible implementation, the hash layer includes a multilayer perceptron and an activation function.

[0097] Specifically, the outputs of the semantic interaction module are the second image features. Second text features The second image feature and the second text feature are respectively input into the corresponding hash layer. In the hash layer, the input is processed by a multilayer perceptron, and then Tanh is used as the activation function to map them into K-bit image hash features and text hash features, thus obtaining the second image hash feature. Second text hash features These are the continuous real-valued feature vectors of the image and text obtained after semantic interaction. The specific process is as follows:

[0098]

[0099]

[0100] In the formula, MLP represents a multilayer perceptron.

[0101] However, cross-modal hash retrieval requires performing a hash function on the image or text to obtain a hash code. In this process, information from the other modality cannot be leaked. Therefore, it is necessary to generate image hash features from image features, text hash features from text features, and generate hash codes.

[0102] Similarly, the first image feature f I and the first text feature f T Input the corresponding hash layers respectively to obtain the first image hash features. and the first text hash feature Generate image hash code based on the first image hash feature and the first text hash feature. and text hash code The expression is as follows:

[0103]

[0104]

[0105]

[0106]

[0107] In the formula, sign represents the sign function, which generates the final binary code. For an image hash code of length K, It is a text hash code of length K.

[0108] In one possible implementation, after obtaining the image hash code and the text hash code, the following steps are included:

[0109] The similarity between different instances is obtained by using the image hash code and text hash code obtained from different instances.

[0110] Specifically, in Hamming space, the similarity between two hash codes is calculated using Hamming distance, which is obtained by calculating the inner product of the two hash codes. In the formula, i ,b j > represents the inner product of two hash codes, where K is the length of the hash code.

[0111] The relevance between images and text is described by a similarity matrix S between image modalities and text modalities. In the experiment, when two training instances are o i and o j When two instances share one or more tags, they are considered similar, i.e., S ij =1, given a similarity matrix S, the probability of the predicted matrix under a known hash code B is expressed as:

[0112]

[0113] In the formula, Compared to two dissimilar instances, two similar instances should have a higher probability result after the inner product. During the evaluation phase, the final binary code is generated by applying a sign function, as follows:

[0114] b = sign(h) ∈ {-1, 1} K

[0115] In the formula, h is the hash feature generated after the instance passes through the hash network function.

[0116] 105. Obtain the target loss based on the image hash code and the text hash code, and update the initial cross-modal hash model according to the target loss to construct a cross-modal hash model.

[0117] In one possible implementation, obtaining the target loss based on the image hash code and the text hash code includes:

[0118] The target loss is obtained based on the image hash code, the text hash code, the second image feature, the second text feature, the first image hash feature, the second image hash feature, the first text hash feature, and the second text hash feature.

[0119] In one possible implementation, obtaining the target loss based on the image hash code, the text hash code, the second image feature, the second text feature, the first image hash feature, the second image hash feature, the first text hash feature, and the second text hash feature includes:

[0120] ​Based on the second image features and the second text features, a cross-modal semantic interaction feature triplet loss is obtained;

[0121] The hash feature triplet loss is obtained based on the second image hash feature and the second text hash feature;

[0122] The feature alignment loss is obtained based on the second image feature, the second text feature, the first image hash feature, the second image hash feature, the first text hash feature, and the second text hash feature;

[0123] The quantization loss is obtained based on the image hash code, the text hash code, the first image hash feature, and the first text hash feature;

[0124] The target loss is obtained based on the cross-modal semantic interaction feature triplet loss, the hash feature triplet loss, the feature alignment loss, and the quantization loss.

[0125] Specifically, pairwise losses are frequently used in many methods to preserve the similarity structure between instances. However, pairwise losses do not guarantee higher-level relationships of similarity among all features. Therefore, cross-modal semantic interaction feature triplet loss uses cross-modal cosine triplet loss to reduce the differences between modalities. Taking the image modality as an example, the triplet takes the form of: Here It is the second image feature. and It is a text feature, and text features Semantically related to the second image features Similar to, and with Conversely, the cosine triplet loss means using cosine distance to measure the similarity between items. The triplet loss for the image modality is expressed as:

[0126]

[0127] In the formula, m is the margin parameter, and L I→T It is the triplet loss of the image modality.

[0128] For the text modality, the triplet loss is expressed as:

[0129]

[0130] In the formula, L T→I The triplet loss representing the text modality. As a second text feature, and Both represent image features, where, Semantically similar to second text features Similar to, and with on the contrary.

[0131] Therefore, the cross-modal semantic interaction feature triplet loss is:

[0132] L ia =L I→T +L T→I

[0133] In the formula, L ia This represents the loss of cross-modal semantic interaction feature triples.

[0134] The hash feature triplet loss constrains the hash features through triplet loss, reducing the heterogeneity gap, as shown below:

[0135] L h =L′ I→T +L′ T→I

[0136]

[0137]

[0138] In the formula, L h L′ represents the hash feature triplet loss. I→T L′ represents the hash feature triplet loss of the image modality. T→I The hash feature triplet loss represents the text modality. Indicates the hash feature of the second image. and Both represent text hash features. Semantically related to the second image hash feature Similar to, and with on the contrary. Indicates the second text hash feature. and Both represent image hash features. Semantically related to the second text hash feature Similar to, and with on the contrary.

[0139] To avoid revealing information from another modality when generating image-text hash codes, the first image feature f is used. I and the first text feature f T To generate hash features and hash codes, and to ensure that the hash codes generated by a single modality take into account the other modality as much as possible, the first image features and the first text features need to be as close as possible to the second image features processed by the semantic interaction module. Second text features Simultaneously, a hash layer with shared weights is used to generate hash features, and the hash features of the first image are guaranteed. and the first text hash feature As close as possible to the second image hash features processed by the semantic interaction module Second text hash features Therefore, the design feature alignment loss is:

[0140] L AL =L FA +L HA

[0141]

[0142]

[0143] In the formula, L AL L represents the feature alignment loss. FA L represents the semantic interaction feature alignment loss. HA f represents the hash feature alignment loss. v f represents the second image label feature. t Represents the second text label feature, |||| F Represents the norm.

[0144] To minimize the quantization loss after sign function quantization when generating hash codes from image hash features and text hash features, the quantization loss is set as follows:

[0145]

[0146] In the formula, L quan b represents the quantized loss. I h represents the image hash code. I Represents the image hash feature, b T Represents the text hash code, h T This represents the text hash feature.

[0147] The total target loss is:

[0148] L total =αL ia +βL h +γL AL +L quan

[0149] In the formula, L total The target loss is represented by α, β, and γ, which are balance parameters.

[0150] Another embodiment of the cross-modal hash retrieval method based on deep semantic interaction of the present invention includes the following steps:

[0151] The data to be retrieved is input into the cross-modal hash model established by the cross-modal hash model construction method based on deep semantic interaction as described above, and the retrieval results are obtained.

[0152] The search results are the data that is closest to the data to be searched. The data to be searched can be either an image or text. If the data to be searched is an image, the search results will be text; if the data to be searched is text, the search results will be images.

[0153] The proposed cross-modal hash retrieval method based on deep semantic interaction is executed on three commonly used cross-modal multi-label databases, namely MIRFlickr25K, NUS-WIDE and MSCOCO, and compared with a number of state-of-the-art cross-modal hash retrieval methods, including DCMH, SSAH, AGAH, DADH and HMAH, to demonstrate the effectiveness of the proposed method.

[0154] The selected evaluation metrics are: topN accuracy curve (topN), precision-recall curve (PR), and mean precision (MAP). Experimental results demonstrate the superiority of this method.

[0155] The selected comparison methods are as follows:

[0156] DCMH, Deep Cross-Modal Hashing, is an end-to-end learning framework where each modality uses a neural network to perform feature learning and hash learning processes from scratch, mapping images and text to Hamming space to generate high-quality hash codes.

[0157] SSAH, Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrieval, consists of three parts: LabNet, a self-supervised semantic generation network; ImgNet, an image adversarial network; and TexNet, a text adversarial network. LabNet learns semantic features from multi-label annotations and combines them with the two adversarial networks, thereby effectively preserving the semantic relevance and consistency of feature distribution between different modalities.

[0158] AGAH, Adversary Guided Asymmetric Hashing for Cross-Modal Retrieval, enhances feature extraction capabilities using a multi-label attention module, while maintaining modality invariance during feature learning using a discriminator and preserving multi-label semantic information for hash codes using asymmetric hashing methods.

[0159] DADH, Deep Adversarial Discrete Hashing for Cross-Modal Retrieval, uses adversarial loss to guide features from both modalities in both the feature learning and hash learning phases to maintain modality invariance, and uses weighted cosine triplet loss to maintain the similarity of image-text pairs.

[0160] HMAH, Teacher-Student Learning: Efficient Hierarchical Message Aggregation Hashing for Cross-Modal Retrieval, proposes a cross-modal relevance knowledge distillation strategy that seamlessly transfers fine-grained multimodal semantic associations from the teacher network to a lightweight student network.

[0161] The experimental parameters and environment are set as follows. The experimental settings for the three datasets used in this chapter are shown in Table 4 below. The hash code lengths are set to 16 bits, 32 bits, and 64 bits, respectively.

[0162] Table 4

[0163] Dataset MIRFlickr25K NUS-WIDE MSCOCO Search set 18015 193734 117218 training set 10000 10500 10000 test set 2000 2100 5000

[0164] For the image modality, the initial weights of the CNN-F network used are parameters from a pre-trained model learned from the ImageNet dataset. For the text modality, Gaussian random initialization is used with all parameters having a mean of 0 and a standard deviation of 0.01.

[0165] The learning rate is initialized to 1e-3, and the learning rate is reduced to one-tenth of its original value every 100 iterations. The maximum number of iterations is set to 300, and the batch size is set to 128.

[0166] During training, our model was trained using the Adam optimizer until it converged. On the MIRFlickr25k dataset, the parameters were set to α = 2, β = 1, and γ = 10. On the NUSWIDE dataset, the parameters were set to α = 2, β = 1, and γ = 20. On the MSCOCO dataset, the parameters were set to α = 2, β = 1, and γ = 20.

[0167] The cross-modal hashing method based on deep semantic interaction proposed in this invention is developed based on the PyTorch deep framework. All applications are run on servers manufactured by NVIDIA, equipped with two graphics cards, namely TITAN X (Pascal) and TITAN XP, and an i7-7700K processor.

[0168] For the MIRFlickr25k dataset, the input images are preprocessed and used as input to the image modality network. The input image size is 224×224×3. The text network input is 1386-dimensional bag-of-words vector features.

[0169] For the NUS-WIDE dataset, the input images are preprocessed and used as input to the image modality network. The input image size is 224×224×3. The text network input is 1000-dimensional bag-of-words vector features.

[0170] For the MS COCO dataset, the input image is preprocessed and used as input to the image modality network. The input image size is 224×224×3. The text network input is a 2000-dimensional bag-of-words vector feature.

[0171] Experimental Results and Analysis:

[0172] Table 5 below compares the MAP values ​​of the method provided in this embodiment of the invention, namely the DSISH (Deep semantic interaction supervised cross-modal hashing) method, with all the comparison methods on the MIRFlickr25k dataset, where the hash code lengths are set to 16 bits, 32 bits, and 64 bits, respectively.

[0173] In all experiments, two cross-modal hash retrieval tasks were set up: image retrieval of text and text retrieval of image.

[0174] Table 5

[0175]

[0176] As can be seen from Table 5, on the MIRFlickr25k dataset, the DSISH method achieved the best results in both image retrieval of text and text retrieval of images.

[0177] Table 6

[0178]

[0179] Table 6 compares the MAP values ​​of the proposed DSISH method with all the contrasting methods on the NUS-WIDE dataset, where the hash code lengths are set to 16 bits, 32 bits, and 64 bits, respectively. All experiments included two cross-modal hash retrieval tasks: image-to-text retrieval and text-to-image retrieval.

[0180] Table 7 below compares the MAP values ​​of the DSISH method with all the comparison methods on the MS COCO dataset.

[0181] Table 7

[0182]

[0183] In all experiments, two cross-modal hash retrieval tasks were set up: image retrieval of text and text retrieval of image.

[0184] In this embodiment of the invention, tag features are extracted, and the feature interaction module is used to fully semantically interact image features and text features under the action of tag features, thereby improving the semantic representation of each modality and generating higher quality hash codes, thus improving the accuracy of cross-modal hash retrieval.

[0185] This article uses specific embodiments to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for constructing a cross-modal hash model based on deep semantic interaction, characterized in that, include: Obtain training instances; The training instances include image-text pairs and multiple labels for the image-text pairs; The image-text pair and the label are input into the feature extraction module of the initial cross-modal hashing model to obtain the first image feature, the first text feature, and the label feature; The first image feature, the first text feature, and the label feature are input into the feature interaction module in the initial cross-modal hashing model to perform feature interaction and obtain the second image feature and the second text feature. The first image feature, the first text feature, the second image feature, and the second text feature are input into the hash generation module in the initial cross-modal hash model to obtain the image hash code and the text hash code; The target loss is obtained based on the image hash code and the text hash code. The initial cross-modal hash model is updated according to the target loss to construct a cross-modal hash model. The step of inputting the first image feature, the first text feature, and the label feature into the feature interaction module in the initial cross-modal hash model for feature interaction to obtain the second image feature and the second text feature includes: The first image feature and the label feature are concatenated to obtain the first image label feature; The first image label features are input into the transformer encoder to obtain the second image label features; The first text feature and the label feature are concatenated to obtain the first text label feature; The first text label features are input into the transformer encoder to obtain the second text label features; The second image feature and the second text feature are obtained by semantically interacting with the second image label feature and the second text label feature through the semantic interaction module. The step of inputting the first image feature, the first text feature, the second image feature, and the second text feature into the hash generation module in the initial cross-modal hash model to obtain the image hash code and the text hash code includes: The first image feature, the first text feature, the second image feature, and the second text feature are respectively input into the corresponding hash layers to obtain the first image hash feature, the first text hash feature, the second image hash feature, and the second text hash feature; the hash layer corresponding to the first image feature and the hash layer corresponding to the second image feature share the same weight; the hash layer corresponding to the first text feature and the hash layer corresponding to the second text feature share the same weight. The image hash code and the text hash code are generated based on the first image hash feature and the first text hash feature, respectively.

2. The method for constructing a cross-modal hash model based on deep semantic interaction as described in claim 1, characterized in that, The step of inputting the image-text pair and the label into the feature extraction module of the initial cross-modal hashing model to obtain the first image feature, the first text feature, and the label feature includes: The image in the image-text pair is input into a CNN-F network to obtain the first image feature; The text in the image-text pair is input into a multilayer perceptron to obtain the first text feature; The label is input into a multilayer perceptron to obtain the label features.

3. The method for constructing a cross-modal hash model based on deep semantic interaction as described in claim 1, characterized in that, The step of performing semantic interaction between the second image label features and the second text label features through the semantic interaction module to obtain the second image features and the second text features includes: The second image label feature and the second text label feature are concatenated to obtain the image-text concatenation feature; The image text concatenation features are input into the transformer encoder to obtain the output of the transformer encoder; The output of the transformer encoder is input into the feedforward network to obtain the second image features and the second text features.

4. The method for constructing a cross-modal hash model based on deep semantic interaction as described in claim 1, characterized in that, The hash layer includes a multilayer perceptron and an activation function.

5. The method for constructing a cross-modal hash model based on deep semantic interaction as described in claim 1, characterized in that, After obtaining the image hash code and text hash code, the process includes: The similarity between different instances is obtained by using the image hash code and text hash code obtained from different instances.

6. The method for constructing a cross-modal hash model based on deep semantic interaction as described in claim 1, characterized in that, The method of obtaining the target loss based on the image hash code and the text hash code includes: The target loss is obtained based on the image hash code, the text hash code, the second image feature, the second text feature, the first image hash feature, the second image hash feature, the first text hash feature, and the second text hash feature.

7. The method for constructing a cross-modal hash model based on deep semantic interaction as described in claim 6, characterized in that, The step of obtaining the target loss based on the image hash code, the text hash code, the second image feature, the second text feature, the first image hash feature, the second image hash feature, the first text hash feature, and the second text hash feature includes: Based on the second image features and the second text features, a cross-modal semantic interaction feature triplet loss is obtained; The hash feature triplet loss is obtained based on the second image hash feature and the second text hash feature; The feature alignment loss is obtained based on the second image feature, the second text feature, the first image hash feature, the second image hash feature, the first text hash feature, and the second text hash feature; The quantization loss is obtained based on the image hash code, the text hash code, the first image hash feature, and the first text hash feature; The target loss is obtained based on the cross-modal semantic interaction feature triplet loss, the hash feature triplet loss, the feature alignment loss, and the quantization loss.

8. A cross-modal hash retrieval method based on deep semantic interaction, characterized in that, The method includes: The data to be retrieved is input into the cross-modal hash model established by the cross-modal hash model construction method based on deep semantic interaction as described in any one of claims 1-7, and the retrieval results are obtained.

Images