Image-text retrieval method and system based on cross-attention hashing network

Abstract

The present disclosure proposes a method and system for image-text retrieval based on a cross-attention hash network, the method comprising the following steps: obtaining input images and texts, and performing feature extraction; performing cross-attention on the extracted features, identifying segments related to shared semantics and deleting irrelevant segments, and obtaining image features with text attention and text features with image attention; performing hash learning on the image and text features with cross-attention respectively to obtain the optimal binary hash code, and then performing similarity measurement to obtain retrieval results of different modalities. The present disclosure identifies segments related to shared semantics and deletes irrelevant segments during the cross-attention process, which can avoid interference from irrelevant segments that lead to semantic dislocation and deviation problems, can improve the accuracy of retrieval, and at the same time reduce the amount of post-processing data and improve retrieval speed.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to data retrieval, and in particular, to a method and system for image and text retrieval based on a cross-attention hash network. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the rapid growth of cross-modal data (such as text, images, and audio) on the Internet, cross-modal retrieval has attracted increasing attention due to its application in processing large-scale media data. The key issue of cross-modal retrieval is to solve the "semantic gap" between different modalities, that is, to find shared semantics with similarities between two modalities in the underlying features of different modalities.

[0004] The latest research has demonstrated the effectiveness of cross-modal hashing retrieval, as it has the advantages of fast retrieval speed and low storage cost. The purpose of cross-modal hashing is to learn binary hash codes of samples from different modalities. Compared with classical hashing methods, cross-modal hashing needs to consider the differences between different modalities. Due to the heterogeneity between different modalities, capturing the semantic relevance between different modalities remains a challenging problem.

[0005] Most existing cross-modal hashing methods capture the semantic relevance of different modalities in a common Hamming space. Supervised cross-modal hashing methods can use semantic labels or related information for supervised learning to extract the semantic relevance between different modalities and obtain better retrieval performance. However, these existing shallow cross-modal hashing methods are almost all based on features produced by traditional manual methods, such as color features, texture features, etc. This may limit the feature representation of the modality, thereby reducing the accuracy of the learned binary hash code.

[0006] In recent years, with the remarkable achievements of deep learning in different fields, existing cross-modal hashing methods have also applied it to the extraction of deep features. For example, Deep Cross-Modal Hashing (DCMH) puts the process of feature extraction and hash code acquisition in the same architecture for training, forming an end-to-end learning architecture, directly generating binary codes through discrete optimization, and improving the retrieval performance to a certain extent. Subsequently, driven by the recent progress in other cross-modal applications such as visual question answering and image captioning, attention has become an important component in the image-text retrieval framework because it allows less attention to unimportant fragments and more attention to important fragments in terms of specific semantics. Some attention-based methods process fragments from different modalities in parallel. However, these methods follow an invariant attention framework, which is to discover shared semantics by paying different attention to all fragments.

[0007] The inventors found that although shared semantics can be found in existing attention mechanisms, they cannot accurately reflect related fragments. This is because many fragments are irrelevant to the shared semantics, but the shared semantics will still receive attention, so that the shared semantics will be more or less disturbed. Therefore, when learning to associate shared semantics selected from images and texts, it will cause semantic misalignment, that is, unrelated fragments from different modalities are closely related except for related fragments. Given a text fragment "Drum", traditional attention methods not only focus on the target image area, but also on other irrelevant areas, such as stools, people, and electronic pianos, which will mistakenly increase the correlation between "Drum" and these irrelevant areas during training, resulting in inaccurate output retrieval results. Summary of the invention

[0008] In order to solve the above problems, the present disclosure proposes a graph-text retrieval method and system based on a cross-attention hash network, which deletes irrelevant fragments in the cross-attention process and focuses on relevant fragments to improve the accuracy of retrieval learning.

[0009] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0010] One or more embodiments provide a method for image-text retrieval based on a cross-attention hash network, comprising the following steps:

[0011] Get input images and text and perform feature extraction;

[0012] It is used to perform cross-attention on the extracted features, identify segments related to shared semantics and delete irrelevant segments, and obtain image features with text attention and text features with image attention;

[0013] The image and text features with cross attention are hashed to obtain the optimal binary hash code, and then similarity measurement is performed to obtain retrieval results of different modalities.

[0014] One or more embodiments provide a graph-text retrieval system based on a cross-attention hash network, including:

[0015] A feature extraction module is configured to obtain input images and texts and perform feature extraction;

[0016] a cross-attention module configured to perform cross-attention on the extracted features, identify segments related to shared semantics and delete irrelevant segments, and obtain image features with text attention and text features with image attention;

[0017] The hash learning module is configured to perform hash learning on the image and text features with cross attention respectively to obtain the optimal binary hash code, and then perform similarity measurement to obtain retrieval results of different modalities.

[0018] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps described in the above method are completed.

[0019] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps described in the above method are completed.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention identifies fragments related to shared semantics and deletes irrelevant fragments during the cross-attention process of features, which can avoid interference from irrelevant fragments that may lead to semantic dislocation and deviation problems, improve the accuracy of retrieval, and at the same time reduce the amount of post-processing data and improve the retrieval speed.

[0022] The advantages of the present disclosure and the advantages of additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and the description thereof are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a method flow chart of Embodiment 1 of the present disclosure;

[0025] Figure 2 is a schematic diagram of a method execution process of a method application example of Embodiment 1 of the present disclosure;

[0026] Figure 3 is a comparison diagram of the instance retrieval result in the method of Embodiment 1 of the present disclosure and the recognition result of an existing method;

[0027] FIG4( a ) is a PR curve diagram on the IAPR TC-12 data set in the experimental results of Example 1 of the present disclosure;

[0028] FIG4( b ) is a PR curve diagram on the MIR-Flickr25k dataset in the experimental results of Example 1 of the present disclosure;

[0029] FIG4( c ) is a PR curve diagram on the NUS-WIDE dataset in the experimental results of Example 1 of the present disclosure;

[0030] Figure 5 This is a comparison chart of the results of the experimental text-to-image retrieval of Example 1 of the present disclosure. DETAILED DESCRIPTION

[0031] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.

[0034] Example 1

[0035] In the technical solutions disclosed in one or more embodiments, Figure 1-3 As shown in FIG. 1 , the image-text retrieval method based on the cross-attention hash network includes the following steps:

[0036] Step 1, feature extraction: obtain input image V and text T and perform feature extraction;

[0037] Step 2, cross attention: The extracted features are cross-attended using the constructed cross attention module to identify segments related to shared semantics and delete irrelevant segments, thus obtaining image features with text attention and text features with image attention;

[0038] Step 3, hash learning: Input the image and text features with cross-attention into the hash learning module for hash learning to obtain the optimal binary hash code, and then perform similarity measurement to obtain the retrieval results and retrieve data of different modalities.

[0039] In this embodiment, the features are cross-attentioned to identify fragments related to shared semantics and delete irrelevant fragments, which can avoid interference from irrelevant fragments and cause semantic dislocation and deviation problems, improve the accuracy of retrieval, reduce the amount of post-processing data, and improve the retrieval speed.

[0040] like Figure 2 , which is a flowchart of a specific example of executing the method of this embodiment, wherein the text is "Four women dressed in orange sing into microphones and play drums". The following will be described in detail with reference to the example.

[0041] In step 1, feature extraction includes image feature extraction and text feature extraction. The input image V and text T are obtained, and the image and text are represented as a set of regional features V = {v1,...,v m} and a set of words T = {t1,...,t n}.

[0042] In some embodiments, the image feature extraction part can use DenseNet, which reuses features by establishing dense connections between all the front layers and the back layers in the network, greatly reducing the number of network parameters, strengthening feature propagation, and effectively solving the gradient vanishing problem.

[0043] Optionally, each image is composed of multiple different regions, and the method for extracting image features, specifically, uses an object detection tool to detect some significant regions in the image region that have prominent semantics for the whole image, and encodes each significant region into a feature vector.

[0044] Among them, the salient areas with prominent semantics are called target areas. The information expressed by an image is generally concentrated on a certain point in the image. It may be a person, a building or even a bucket. For example, an image of a bird will focus on the bird in the image, or other interesting parts, such as a tree or other small animals in the image.

[0045] In this embodiment, specifically, DenseNet (Dense Convolutional Network, translated as densely connected network) is used to detect image areas and extract image features.

[0046] In this embodiment, DenseNet is composed of multiple dense blocks and multiple transition layers, wherein the dense blocks are composed of multiple convolution blocks connected alternately, and each convolution block connects the output feature maps of all previous layers as its input, and then transmits its output to all subsequent layers, which can maximize the information exchange between all layers in the network, greatly reduce the number of network parameters, strengthen feature propagation, and solve the gradient disappearance problem. The transition layer is used to control the number of channels.

[0047] Optionally, the DenseNet model structure used in this embodiment consists of 4 dense blocks and 3 transition layers.

[0048] Specifically, DenseNet is used to predict the region bounding box and score it, the top k significant regions are selected according to the score, and the convolutional model is used to extract the convolutional features f i , and the convolutional features f are transformed through the fully connected layer i Convert to feature vector V i ,as follows:

[0049] v i =W v f i +b v ,i∈[1,m] (1)

[0050] Among them, W v is the learning parameter matrix, b v is the bias vector. Finally, the image V is represented as {v1,v2,...,v m},V∈R d×m , d is the dimension of the image feature vector, and m refers to the number of regions in the image.

[0051] In this embodiment, k is set to 36, and the pre-trained ResNet-101 model is used to extract the 2048-dimensional average pooling convolution features f of these regions. i,i∈[1,m] , which represents each selected region. The final fully connected layer is used to transform f i Convert to a 1024-dimensional feature vector.

[0052] In some embodiments, the text feature extraction part can use the Bi-GRU algorithm, which can effectively suppress the gradient disappearance or explosion when capturing long sequence semantic associations. The effect is better than the traditional RNN and the computational complexity is small.

[0053] Bi-GRU (Bi-directional Gated Recurrent Unit) is a deep learning model that can be used as a framework for semantic parsing in the field of natural language processing.

[0054] Each text contains a set of words. Each word is encoded into a d-dimensional feature vector like the image region, and they are combined into the global features of the text. To this end, Bi-GRU is used to integrate forward and backward context information into word representation. Specifically, the method of extracting text features using the Bi-GRU algorithm includes the following steps:

[0055] Step 1-1, split the acquired text into multiple words, encode each word into a vector, and process it into a set dimension;

[0056] Specifically, the text T is first divided into multiple words, and each word is encoded into a one-hot vector g j ,j∈[1,n], n is the number of words in the text T. In order to reduce the computational cost of GRU, g j Through the parameter matrix W g and the bias vector b g Embedded into a vector o of set dimension j In , the dimension can be set to 300.

[0057] o j =W g g j +b g ,j∈[1,n] (2)

[0058] In step 1-2, the encoded vector is processed using the Bi-GRU algorithm model, the forward and backward context information of the corresponding word vector is extracted, and the average value of the forward and reverse hidden states is calculated as the text representation to obtain the text feature vector.

[0059] Specifically, the encoded vector is the low-dimensional vector o j It is fed into the Bi-GRU in order to integrate the forward and backward contextual information into the word representation. After multiple iterations, the average of the forward and backward hidden states can be regarded as a text representation, which contains the d-dimensional features of each word in the text.

[0060]

[0061] Among them, tj is a feature vector, and the text T is represented as {t1,t2,...,t n},T∈R d×n , d is the dimension of the text feature vector.

[0062] In this embodiment, each image in a large-scale data set is composed of multiple different regions. The object detection tool is used to detect some significant regions in these regions that have prominent semantics for the whole world, and each significant region is encoded as a feature vector. To this end, DenseNet (Dense Convolutional Network) is used to detect image regions, and Bi-GRU is used to integrate forward and backward context information into word representation.

[0063] In step 2, by removing irrelevant regions, the semantic misalignment and deviation problems of traditional attention can be solved. The constructed cross-attention module is configured to evaluate the relevance of related fragments according to the constructed scoring function, filter relevant fragments related to shared semantics by relevance, and discard irrelevant fragments;

[0064] The scoring function F is: the importance of the compared segment i compared to other segments j (x i ,x j ) and the confidence p(x) of other comparison fragments j j ) is the weighted sum of the products of .

[0065] After using a deep neural network to project each image region and text word into a common d-dimensional space, we get the image representation {v1,v2,...,v m},V∈R d×m and the text representation {t1,t2,...,t n},T∈R d×n For a general hash retrieval network with attention, the attention mechanism is used to learn the attention distribution on all image regions and text words, where regions or word fragments related to shared semantics will receive more attention, while irrelevant regions or word fragments will receive relatively less attention. Finally, the kth shared semantics selected from the image and text is represented by weighted summation. and

[0066]

[0067]

[0068] Among them, w ik and w jk are the attention distribution about the kth shared semantics, v irepresents the i-th image region, t j represents the jth word.

[0069] The above general attention framework for finding shared semantics has certain problems: not all fragments support a specific shared semantics because many of them are irrelevant to the specific shared semantics. If they are aggregated, the shared semantics will be more or less disturbed by irrelevant fragments, resulting in semantic misalignment and deviation problems, because different semantics cannot be properly decoupled.

[0070] In this embodiment, optionally, the relevance of the segments is determined by calculating the importance of the segments relative to other segments. The formula of the scoring function F is as follows:

[0071]

[0072] Among them, the pair function f(x i ,x j ) represents the calculation of the importance of the i-th segment compared to the j-th segment, and p(x j ) represents the confidence of the jth segment being compared.

[0073] For the proposed scoring function, the scores are judged, and the fragments with scores greater than zero are set as relevant fragments, and the subset of relevant fragments is selected using the indicator function, that is:

[0074] H(x)=I(F(x)>0) (6)

[0075] where x can be a region or a word, and I(·) is an indicator function representing the subset of segments that are semantically relevant to the target.

[0076] The scoring function is applied to formula (4) to improve the general attention framework. The weighted comparison results in the general attention framework are added to other comparison fragments. If the fragment is similar to other related fragments with high confidence scores, it can be considered that the fragment is related to the shared semantics. Therefore, the kth shared semantics can be defined as:

[0077]

[0078] This embodiment eliminates irrelevant fragments from the context of image and text modalities, thereby eliminating the interference of irrelevant fragments on shared semantics and solving the problems of semantic dislocation and deviation. In addition, it is completely different from hardattention that uses random methods to estimate gradients. After discarding irrelevant fragments, the cross attention of this embodiment makes the remaining relevant fragments more conducive to forward propagation, and the gradient can be directly calculated, so that network training can be more efficient. The following will apply the improved attention module to image and text modalities for cross attention.

[0079] (1) Cross-Attention from Image to Text.

[0080] Optionally, this embodiment discovers shared semantics in images and text by fixing one modality and finding related segments from another modality, where the segments in the fixed modality are considered as shared semantics. For the image-to-text direction, image regions are fixed as shared semantics, and related text words need to be found for each image region.

[0081] Optionally, the image-to-text cross-attention includes the following steps:

[0082] Step 21: Pre-assign attention: Pre-assign attention weights to each word of the text.

[0083] Specifically, the attention weights are pre-assigned. Specifically, the similarity score between each image region and the word is calculated by using cosine similarity, and the similarity score of each word relative to the query region is normalized to a value within [0, 1] using softmax, as shown in formula (8). The attention weight of the jth word in the i image region is represented as w ij In this process, relevant words can get more attention, while irrelevant words also contribute to the semantic sharing between image regions and target text.

[0084]

[0085] Among them, σ represents the softmax activation function. α is a set scaling factor to further increase the gap between relevant and irrelevant regions, which can be set to 20.

[0086] Step 22, determine related words: score each word according to the pre-allocated attention of each word relative to other words according to the scoring function, and determine the related words of each word. That is:

[0087]

[0088] Among them, f(t ij ,t iq ) represents the importance of the jth word relative to the qth word. The confidence score p(t iq ) is set to its correlation with the i-th region, for example Additionally, the confidence of each text word can be set equal.

[0089] Then, the relevant words are selected by element-wise multiplying each word by the indicator function H of Formula 6. The fragments with scores greater than zero are relevant words, otherwise they are irrelevant words.

[0090] Step 23, reallocate attention: Redistribute attention to these related words by renormalizing, and irrelevant words will not participate in this process because their scores are 0.

[0091]

[0092] Step 24: By performing a weighted summation of the reallocated attention and the word representation of the text in the d-dimensional space, the image feature with text attention is obtained, that is, the shared semantics of the i-th region selected from the text: The learned cross attention w′ ij Determines the weight.

[0093] (2) Attention from text to image

[0094] Similarly, for the text-to-image direction, text words are fixed to share semantics and the relevant image regions need to be found for each text word.

[0095] Step 2-1: Pre-assign attention scores to each region. Specifically, this is achieved by calculating the cosine similarity between the region and the word and normalizing it using the softmax activation function:

[0096]

[0097] Step 2-2, determine the relevant area. Specifically, score each area according to the pre-allocated attention of each area relative to other areas, and then select the relevant area by multiplying the elements between each image area and the function H. The area with a score greater than zero is the relevant area, otherwise it is an irrelevant area.

[0098]

[0099] Step 2-3, reassign the attention weights to these selected relevant regions by renormalization. Among them, irrelevant regions will not participate in this process because their scores are zero:

[0100]

[0101] In step 2-4, the text features with image attention are obtained by weighted summing the reallocated attention weights and the image representation in the d-dimensional space.

[0102] The reallocated attention weight w′ ji Will replace the original attention w in formula (4) jkSince the attention weight of irrelevant regions is zero, attention will be focused on the most relevant regions. The final text feature with image attention, that is, the shared semantics selected from the image according to the jth word, can be calculated as

[0103] In this embodiment, in order to solve the semantic dislocation and deviation problems caused by the general attention framework, starting from the relevance selection of shared semantics, the shared semantics is represented by integrating a subset of fragments related to the target semantics, and a learning scoring function is proposed to identify fragments related to the shared semantics and delete fragments that are not related to the shared semantics. There are also some attention methods that have been used to identify fragments related to shared semantics. They simply mask the fragments according to their positions, but the semantic relevance is independent of the fragment positions. This embodiment determines the relevance of the fragments by calculating the importance of the fragments relative to other fragments, thereby improving the accuracy of recognition.

[0104] In step 3, the similarity is finally measured by Hamming distance to retrieve data of different modalities. The steps include:

[0105] Step 31: Input the obtained cross-attention image and text features into the hash learning module to obtain their binary hash codes. The hash learning module can perform hash learning using a linear hash function, and the formula is as follows:

[0106] Hash v =sgn(w T v+b)

[0107] Hash t =sgn(w T t+b) (14)

[0108] Among them, b represents the offset, w represents the mapping vector. The sgn function is as follows:

[0109]

[0110] It also includes the construction of a hash code library: input the image-text pairs in the known dataset into the framework to build a hash code library for images and texts. Input the image / text modality to be queried into the hash learning module to obtain its binary hash code.

[0111] Step 32: Calculate the Hamming distance between the hash code of the modality to be queried and the hash code of another modality in the hash code library to perform similarity measurement.

[0112] The image / text modality to be queried is input into the framework to obtain its binary hash code, and then the similarity is measured by calculating the Hamming distance between the hash code of the modality to be queried and the hash code of another modality in the hash code library, and the retrieved other modality is returned in ascending order according to the Hamming distance, so as to obtain the top K results with the highest similarity to the query modality. The Hamming distance calculation formula is

[0113]

[0114] Hash x , Hash y They represent the hash code of the modality to be queried and the hash code of another modality in the hash code library, respectively. Represents XOR operation. The smaller the Hamming distance, the higher the similarity between the two different modes.

[0115] Furthermore, the steps of training the cross attention module and the hash learning module are also included:

[0116] Step S1, feature extraction: obtain the image V1 and text T1 of the original data set, and perform feature extraction;

[0117] Step S2, cross attention: using the constructed cross attention module to perform cross attention, identify segments related to shared semantics and delete irrelevant segments, and obtain image features with text attention and text features with image attention;

[0118] Step S3, hash learning: input the image and text features with cross attention respectively into the hash learning module for hash learning to obtain the optimal binary hash code, and then perform similarity measurement to obtain the retrieval result, and retrieve data of different modalities;

[0119] Step S4: Take the triplet loss as the objective function, calculate the objective function value, iterate until the objective function is within the set range, and output the parameters of the cross-attention module and the hash learning module.

[0120] Optionally, in order to optimize the proposed network, this embodiment adopts triplet loss as the objective function, which is used in the field of cross-modal retrieval to shorten the distance between similar samples and increase the distance between heterogeneous samples, and thus has been proven to maximize the relevance scores of related image-text pairs and minimize the relevance scores of unrelated image-text pairs.

[0121] Given a pair of related image-text, the relevance score of the image-text pair is denoted as S vt :

[0122]

[0123] represents the negative samples when using text to retrieve images, Represents a negative sample when using an image to retrieve text. The relevance score between the negative sample and the text or image is forced to be a fixed magnitude lower than the relevance score between the relevant image-text pair, that is, the relevance score between the negative sample and the given sample or image is:

[0124]

[0125] Here, the fixed amplitude α is set to 0.2. And [x] + ≡max(x,0), if the relevance score of the negative sample is not as large as the relevance score of the relevant image-text pair, the loss is set to zero.

[0126] The search results of the example of this embodiment are as follows: Figure 3 As shown in the figure, the comparison between the existing attention and the attention in this paper. The area of ​​attention is defined by different color boxes, where the number in the upper left corner of the color box represents the weight of the attention. The existing attention will focus on areas that are not related to the text query "Drum", such as stools, people, and electronic pianos. When "Drum" is learned to be similar to irrelevant areas, it will cause semantic misalignment. The attention of the embodiment avoids semantic misalignment by eliminating irrelevant areas.

[0127] In order to illustrate the effect of the method of this embodiment, experiments were conducted to evaluate the proposed image-text retrieval method based on cross-attention hashing network (CAHN for short) on three widely used datasets, and this method was compared and analyzed with several current advanced methods using two evaluation indicators.

[0128] (i) Datasets: NUS-WIDE dataset, MIR-Flickr25K dataset and IAPR TC-12 dataset can be used.

[0129] The NUS-WIDE dataset is a large-scale network image dataset created by a media search laboratory. The dataset contains 260,648 images and 5,018 different class labels collected on the Flickr website. Each image has its corresponding text annotation and constitutes an image-text pair. These texts describing the image are a group of sentences that the user connects with words when uploading the image. In this embodiment, the benchmark method is analyzed based on 194,600 image-text pairs of the 20 most commonly used tags in this dataset, and the text of each pair of data is represented as a 1000-dimensional bag-of-words (BOW) vector. If the image and text have one of the same concept labels, they are considered similar, otherwise they are considered dissimilar.

[0130] The MIR-Flickr25K dataset contains 25,000 multi-label images collected from the Flickr website and 24 manually annotated category labels. This embodiment selects image-text pairs with at least 20 text labels, and a total of 20,015 pairs of data are obtained, each pair of data is labeled with one of the 24 categories of labels. The text of each pair of data is represented as a 1386-dimensional BOW vector. If the image and text have the same label, they are considered similar, otherwise they are considered dissimilar.

[0131] The IAPR TC-12 dataset contains 20,000 images, each with a text caption. The image-text pairs are labeled with 255 labels. For the text modality, a 2912-dimensional BOW vector is used to represent the text features. If the image and text have the same label, they are considered similar, otherwise they are considered dissimilar.

[0132] In terms of data set splitting, this embodiment randomly selects 2,500 pairs of data from the MIR-Flickr25K data set and the IAPR TC-12 data set as query sets, and the remaining pairs of data are used as retrieval sets. In addition, 10,000 pairs of retrieval sets randomly selected are used as their training sets. For the NUS-WIDE data set, 1% of the data set is randomly selected as the query set, and the rest is used as the retrieval set. In addition, 10,500 pairs are randomly selected from the retrieval set as the training set.

[0133] 2. Evaluation indicators

[0134] In order to more carefully evaluate the image-text retrieval performance of the method of this embodiment when the Hamming radius changes, two widely used evaluation indicators are selected: mean Average Precision (mAP) and PR (Precision-Recall) curve.

[0135] mAP is a standard evaluation indicator used to measure the accuracy of Hamming ranking. The calculation formula of mAP value is shown in (19):

[0136]

[0137] Where |Q| represents the size of the query data set Q, and q represents a given query;

[0138] AP stands for Average Precision:

[0139]

[0140] Among them, M represents the number of real neighbors of q in the query data, n represents the total amount of data, and P q(i) represents the accuracy of the first i retrieved instances, δ(i) is an indicator function. When δ(i) = 1, it means that the i-th instance is relevant to the retrieved instance, and when δ(i) = 0, it means it is irrelevant.

[0141] Precision refers to the ratio of the number of modal samples similar to the modal sample to be queried in all retrieval results (TP) to the number of all modal samples in the retrieval results (TP+FP). The comparison range is the final retrieval results, indicating the accuracy of the retrieval. The formula is as follows.

[0142]

[0143] Recall refers to the ratio of the number of samples of another modality similar to the modality sample to be queried in all retrieval results (TP) to the total number of samples of another modality similar to the modality sample to be queried in the dataset (TP+FN). The comparison range is the entire dataset, indicating the comprehensiveness of the retrieval. The formula is as follows.

[0144]

[0145] (III) Parameter settings

[0146] In the experiment, in order to obtain rich feature information, the hyperparameters of the entire model are set, and the dimension of the image region representation is set to d = 1024. The initial one-hot vector of the word embedding is converted to 300 dimensions and then input into the bidirectional GRU to generate a 1024-dimensional representation. The network is trained on the Adam optimizer. The network is trained at a learning rate of 0.0002 on MIRFLICK-25K, a learning rate of 0.0005 on NUS-WIDE, and a learning rate of 0.0003 on IAPR TC-12, and decays by 0.1 after every 10 epochs. The batch size is set to 32. The network of this embodiment is trained for 15 epochs on MIRFLICK-25K, 20 epochs on NUS-WIDE, and 20 epochs on IAPR TC-12, and the training instances are randomly shuffled at each epoch.

[0147] (IV) Comparison results

[0148] In order to verify the performance of the method proposed in this embodiment, the most representative methods are selected as benchmarks for comparison, including shallow cross-modal hashing methods STMH and SePH, and deep cross-modal hashing methods DCMH, AGAH, DDCH, and DSADH.

[0149] Experiments were conducted on the MIRFLICK-25K, NUS-WIDE, and IAPR TC-12 datasets. Table 1 shows the mAP value comparison data of the generated 16, 32, and 64-bit hash codes.

[0150] Table 1

[0151]

[0152] As shown in Figure 4, experiments were conducted on three widely used datasets using hash codes with a coding length of 16 bits. Including the retrieval process from image to text and from text to image, given any Hamming radius, the precision and recall values ​​can be calculated. By changing the Hamming radius between 0 and 16, the PR curve can be obtained. The higher the PR curve position, the better the retrieval performance.

[0153] It can be seen that the CAHN method of this embodiment outperforms all the baseline methods in both subtasks. In particular, for the task of text retrieval from images, the CAHN method is significantly better than other methods. This shows that the improved cross-attention of this embodiment can learn finer-grained region correspondences without the interference of irrelevant regions, so that related segments are closely related, which is of great significance for practical multimodal applications. However, in the task of image retrieval of text, the performance of the CAHN method is slightly worse than that of the supervised cross-modal hashing method. This is because the supervised cross-modal hashing method can embed label and category information into the learning process, which is helpful for the recognition of shared semantics. However, the supervised cross-modal hashing method will affect the overall training speed in the process of embedding other information into the framework, thereby reducing the retrieval effect in reality.

[0154] 5. Ablation studies

[0155] In this example, 10,000 pairs of data randomly selected from the MIR-Flickr25K dataset are used as training sets, and the rest are used as test sets for experiments. In order to verify the effectiveness of the proposed model, an ablation analysis is performed on the model itself.

[0156] In the ablation experiment, a certain module in the model of this study is deleted or replaced while keeping the corresponding parameters unchanged. In order to evaluate the effect of cross attention, all attention is removed in the CAHN model, denoted as CAHN-noAtt, and traditional attention is used in the image-to-text and text-to-image directions, denoted as CAHN-traAtt, and other parameter settings remain unchanged. Table 2 shows the ablation study conducted on the MIR-Flickr25K dataset. It can be seen from the results of each ablation experiment in Table 2 that cross attention is crucial to improving the overall matching performance, especially for the text query image task, because it removes the most irrelevant fragments and avoids the semantic misalignment problem by considering the proportion of relevant fragments rather than their absolute number.

[0157] Table 2

[0158]

[0159]

[0160] (VI) Visualization results

[0161] This experiment shows the visualization results of text-to-image and image-to-text retrieval. Figure 5 The text-to-image retrieval shown in the figure, Query 1, 2, and 3 above are the corresponding text information, and the figure shows the top 3 images for each text query. The images in the first three columns are retrieved by the retrieval method proposed in this embodiment, and the last three columns are retrieved by the benchmark method. The correctly retrieved image is located in the box. Long text and short text queries can be well matched with their most relevant images. For the first text query, although some keywords such as "girl", "boy", and "front" appear in the local area of ​​the second image, the correct image is still ranked first. The baseline model ignores "football" and gives incorrect results because irrelevant areas interfere with semantic alignment. For example, when matching "football", they will notice the irrelevant "football grass field", which will reduce the response to the query term, but the wrong image will give a higher response because "boys and girls are on the football grass field".

[0162] Performance of image retrieval text. Compare the Label on the right side of each image query, the top-ranked text results generated by the method of this embodiment and the baseline method, where the correct results are marked with light-colored characters. As in the first example, the picture is roughly a black and white dog wearing a harness, licking a person wearing glasses. The label Label is: A blackand white dog wearing a harness licks abrunette wearing glasses, and the result recognized by the Baseline algorithm is: "A woman with black eyes adopted a beautiful puppy". The result obtained is not very correct because it always focuses on the keyword "dog", so it ignores some action details in the image (such as "lick" and "harness") when querying. The text output by the method of this embodiment is consistent with the label. This further confirms that using focus attention can capture and distinguish more detailed information. Although most other keywords match the query image, CAHN performs better.

[0163] Example 2

[0164] Based on Example 1, this embodiment provides a graph-text retrieval system based on a cross-attention hash network, including:

[0165] A feature extraction module is configured to obtain input images and texts and perform feature extraction;

[0166] a cross-attention module configured to perform cross-attention on the extracted features, identify segments related to shared semantics and delete irrelevant segments, and obtain image features with text attention and text features with image attention;

[0167] The hash learning module is configured to perform hash learning on the image and text features with cross attention respectively to obtain the optimal binary hash code, and then perform similarity measurement to obtain retrieval results of different modalities.

[0168] Example 3

[0169] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps described in the method of Embodiment 1 are completed.

[0170] Example 4

[0171] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps described in the method of Embodiment 1 are completed.

[0172] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0173] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. The image-text retrieval method based on cross-attention hashing network is characterized by: The steps include: Get input images and text and perform feature extraction; Perform cross-attention on the extracted features, identify segments related to shared semantics and delete irrelevant segments, and obtain image features with text attention and text features with image attention; The image and text features with cross attention are hashed to obtain the optimal binary hash code, and then similarity measurement is performed to obtain retrieval results of different modalities; Perform cross-attention on the extracted features, calculate the relevance of related fragments according to the constructed scoring function, screen related fragments related to shared semantics by relevance, and discard irrelevant fragments; the scoring function is: the weighted sum of the product of the importance of the compared fragment i compared to other fragments j and the confidence of other compared fragments j; A method of performing cross-attention on the extracted features, identifying segments related to shared semantics and removing irrelevant segments, and obtaining image features with text attention and text features with image attention, including cross-attention from image to text and cross-attention from text to image; Image-to-text cross-attention, image regions are fixed to share semantics, and it is necessary to find relevant text words for each image region, including the following steps: Pre-assign attention weights to each word of the text; Scoring each word according to a scoring function based on the pre-assigned attention of each word relative to other words, and determining the related words of each word; Redistribute attention to related words by renormalizing; The reallocated attention and the word representation of the text in the set dimensional space are weighted summed to obtain the image features with text attention; From text to image attention, text words are fixed as shared semantics, and it is necessary to find the relevant image region for each text word, including the following steps: Pre-assign an attention score to each region; According to the pre-allocated attention of each region relative to other regions, each region is scored by a scoring function to determine the relevant regions; Reassign attention weights to these selected relevant regions by renormalizing; The text features with image attention are obtained by weighted summing the reallocated attention weights and the image representation in the set dimensional space.

2. The image-text retrieval method based on cross-attention hashing network as claimed in claim 1, characterized in that: Feature extraction, including image feature extraction and text feature extraction, represents images and texts as a set of regional features and a set of words respectively.

3. The image-text retrieval method based on cross-attention hashing network as claimed in claim 2, characterized in that: Use the DenseNet model to detect image areas and extract image features; The DenseNet model consists of multiple dense blocks and multiple transition layers. The dense blocks are composed of multiple convolutional blocks connected alternately. Each convolutional block connects the output feature maps of all previous layers as its input, and then transmits its output to all subsequent layers. The transition layer is used to control the number of channels.

4. The image-text retrieval method based on cross-attention hashing network as claimed in claim 2, characterized in that: The text feature extraction part uses the Bi-GRU algorithm, which includes the following steps: Split the acquired text into multiple words, encode each word into a vector, and process it into the set dimension; The encoded vector is processed using the Bi-GRU algorithm model to extract the forward and backward context information of the corresponding word vector, and the average value of the forward and reverse hidden states is calculated as the text representation to obtain the text feature vector.

5. The image-text retrieval method based on cross-attention hashing network as claimed in claim 1, characterized in that: The method of performing hash learning on image and text features with cross attention to obtain the optimal binary hash code and then performing similarity measurement to obtain retrieval results of different modalities includes the following steps: The obtained cross-attention image and text features are input into the hash learning module to obtain the corresponding binary hash code; The Hamming distance between the hash code of the modality to be queried and the hash code of another modality in the hash code library is calculated to measure the similarity and obtain the retrieval result.

6. The image-text retrieval method based on cross-attention hashing network as claimed in claim 1, characterized in that: The cross attention module is constructed to realize the cross attention of the extracted features; the optimal binary hash code is obtained by hash learning, and the image and text features with cross attention are input into the hash learning module to obtain the optimal binary hash code by constructing the hash learning module; and the steps of training the cross attention module and the hash learning module are also included: Get the images and text of the original data set and perform feature extraction; Using the constructed cross-attention module, we identify segments related to shared semantics and remove irrelevant segments, and obtain image features with text attention and text features with image attention; The image and text features with cross attention are input into the hash learning module to obtain the optimal binary hash code, and then the similarity measurement is performed to obtain the retrieval results and retrieve data of different modalities; Take the triplet loss as the objective function, calculate the objective function value, iterate until the objective function is within the set range, and output the parameters of the cross-attention module and the hash learning module.

7. The image-text retrieval system based on cross-attention hashing network is characterized by: include: A feature extraction module is configured to obtain input images and texts and perform feature extraction; a cross-attention module configured to perform cross-attention on the extracted features, identify segments related to shared semantics and delete irrelevant segments, and obtain image features with text attention and text features with image attention; The hash learning module is configured to perform hash learning on the image and text features with cross attention to obtain the optimal binary hash code, and then perform similarity measurement to obtain retrieval results of different modalities; Perform cross-attention on the extracted features, calculate the relevance of related fragments according to the constructed scoring function, screen related fragments related to shared semantics by relevance, and discard irrelevant fragments; the scoring function is: the weighted sum of the product of the importance of the compared fragment i compared to other fragments j and the confidence of other compared fragments j; A method of performing cross-attention on the extracted features, identifying segments related to shared semantics and removing irrelevant segments, and obtaining image features with text attention and text features with image attention, including cross-attention from image to text and cross-attention from text to image; Image-to-text cross-attention, image regions are fixed to share semantics, and it is necessary to find relevant text words for each image region, including the following steps: Pre-assign attention weights to each word of the text; Scoring each word according to a scoring function based on the pre-assigned attention of each word relative to other words, and determining the related words of each word; Redistribute attention to related words by renormalizing; The reallocated attention and the word representation of the text in the set dimensional space are weighted summed to obtain the image features with text attention; From text to image attention, text words are fixed as shared semantics, and it is necessary to find the relevant image region for each text word, including the following steps: Pre-assign an attention score to each region; According to the pre-allocated attention of each region relative to other regions, each region is scored by a scoring function to determine the relevant regions; Reassign attention weights to these selected relevant regions by renormalizing; The text features with image attention are obtained by weighted summing the reallocated attention weights and the image representation in the set dimensional space.

8. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps described in any one of the methods of claims 1 to 6 are completed.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps described in any one of claims 1 to 6.

Images