A Hierarchical Alignment-Based Image-Text Retrieval Method

Abstract

The present invention discloses a hierarchical alignment-based image-text retrieval method, which uses a unified Transformer architecture to extract and learn image and text features, and introduces a hierarchical alignment strategy to capture more and richer semantic information at different levels of the Transformer. Therefore, both images and texts adopt a unified Transformer model, and the hierarchy makes full use of the modal features obtained by the Transformer structure. So finally, a better-compatible representation is obtained through a unified model architecture, achieving better semantic alignment and realizing image-text retrieval.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal retrieval. More specifically, it is a hierarchical alignment-based image-text retrieval method. Background Art

[0002] The cross-modal retrieval task is one of the basic and common tasks in the multi-modal field. Its goal is to accurately retrieve the sample corresponding to another modality through a sample of one modality. Here, the retrieval task between the two most important and common modalities, namely visual information and text data, is studied.

[0003] Existing cross-modal retrieval methods can be divided into global alignment methods and local alignment methods. The global alignment method explores the correlation by mapping the image and text as a whole into a unified semantic space for measurement. Usually, a Convolutional Neural Network (CNN) is used to extract image features, and a Recurrent Neural Network (RNN) is used to extract text features. However, such a method ignores the local clues between image regions and text words. Therefore, the local alignment method is proposed, which models by extracting the local features of image text to complete the alignment between image and text. Usually, Faster R-CNN (Faster Region-based Convolutional Neural Network) is used to extract image region features, and an RNN such as Bi-GRU (Bidirectional Gated Recurrent Unit) is used to extract text words. Although the local alignment-based cross-modal retrieval method can effectively utilize the subtle clues between image regions and text words to further complete a better alignment between image and text. However, whether it is local alignment or global alignment, the architectures adopted for different modalities are different during feature extraction. That is, for image data, a convolutional neural network with inductive biases related to images such as multi-scale, locality, and translational invariance is used to extract feature information, while for text data, a recurrent neural network with inductive biases related to language such as temporality is used to extract feature information. Although such a feature extraction method conforms to the characteristics of the corresponding modality information respectively, in the cross-modal retrieval task, the key lies in the modeling and measurement of the common space. Therefore, different model architectures will cause two deficiencies: (1) Different architectures may cause different semantic distribution spaces, making the finally learned representations less compatible in semantic mapping and similarity measurement. (2) By using different architectures for feature extraction, it is impossible to extract the required information comprehensively and from shallow to deep. The corresponding relationships at different semantic levels cannot be captured. Summary of the Invention

[0004] The object of the present invention is to overcome the deficiencies of the prior art and provide a method for hierarchical alignment of image text retrieval. Specifically, a unified structure is adopted to hierarchically make full use of the modal features obtained by the Transformer (deep self-attention transformation model) structure, so as to obtain a better compatible representation and complete better semantic alignment.

[0005] To achieve the above object of the invention, a method for hierarchical alignment of image text retrieval of the present invention is characterized by comprising the following steps:

[0006] (1), Establish a retrieval library;

[0007] Select an image I to be retrieved, and all texts T containing the text matching the image I t are composed into a text set, and finally the image I and the corresponding text set are used as the retrieval library, where t represents the t-th text;

[0008] (2), Extract image text features;

[0009] (2.1), Extract image features;

[0010] Compose 12 Transformer modules into a Swin-Transformer (shift window Transformer) model. The 12 Transformer modules are divided into 3 stages: low, medium, and high. After the Swin-Transformer model is pre-trained, feature extraction of the image regions of the image I is performed to obtain feature output maps of the 3 stages of low, medium, and high, which are respectively denoted as V low , V middle , V high , so as to represent the hierarchical feature V h as:

[0011] V h ={V low , V middle , V high}

[0012]

[0013]

[0014]

[0015] Among them, represents the image features of the k-th, m-th, and g-th image regions divided in the three stages of low, medium, and high. K, M, and G are the number of regions divided in different stages;

[0016] (2.2), Extract text features;

[0017] Compose a BERT (Bidirectional Encoder Representations from Transformers) model with 12 Transformer modules. The 12 Transformer modules are divided into three stages: low, medium, and high. After the pre-training of the BERT model is completed, text feature extraction is performed on all texts to obtain text features T at the low, medium, and high stages t , denoted as W t low ,W t middle ,W t high , thus representing the hierarchical features as:

[0018]

[0019]

[0020]

[0021]

[0022] where l = 1, 2, …, L, represents the l-th word feature extracted from the low, medium, and high stages of the t-th text, and L is the number of words in the t-th text;

[0023] (3), Hierarchical alignment;

[0024] (3.1), Calculate the cross-attention weight S of the p-th image region corresponding to the q-th word in the t-th text pq :

[0025]

[0026] where ||·|| represents the L2 norm operation; generally refers to one of the features; generally refers to one of the features; the superscript T represents transpose;

[0027] (3.2), Based on the attention cross-weight, calculate the weight a of the p-th image region corresponding to the q-th word in the t-th text pq ;

[0028]

[0029] (3.3), Sum the weights of all words in the p-th image region corresponding to the t-th text to obtain the weighted sum word feature a corresponding to the p-th image region p ;

[0030]

[0031] Among them, is the normalized value of S, and λ is the temperature hyperparameter; pq The normalized value of S, λ is the temperature hyperparameter;

[0032] (3.4), Calculate the feature v of the p-th image region p and the cosine similarity cos(v p , a p , a p ) after weighted summation with the corresponding word feature a;

[0033] (3.5), Add the cosine similarities between all the image regions in the same stage among the low, medium, and high stages and the corresponding word features after weighted summation to obtain the similarities of each stage, denoted as S low , S middle , S high

[0034]

[0035] Among them, S ∈ S low , S middle , S high ;

[0036] (3.6), Add the similarities of each stage as the retrieval value of the image I and the t-th text;

[0037] (4), Obtain the image-text retrieval result;

[0038] Traverse all the texts in the retrieval text library, obtain the retrieval values of the image I and each text, arrange all the retrieval values from largest to smallest, and then take the text ranked first as the retrieval result.

[0039] A hierarchical alignment image-text retrieval method of the present invention uses a unified Transformer architecture to extract and learn image and text features, and introduces a hierarchical alignment strategy to capture more and richer semantic information at different levels of the Transformer. Therefore, both images and texts use a unified Transformer model, and the hierarchy makes full use of the modal features obtained by the Transformer structure. So finally, a better compatible representation is obtained through a unified model architecture, better semantic alignment is completed, and image-text retrieval is realized.

[0040] Meanwhile, a hierarchical alignment image-text retrieval method of the present invention also has the following beneficial effects:

[0041] (1) Compared with the original traditional method, this patent adopts a unified Transformer architecture to extract and learn image and text features. With the same model architecture, it can learn more compatible modal features in the semantic space and achieve better semantic alignment between modalities.

[0042] (2) Since the same Transformer model architecture is used for both image and text feature extraction, the present invention introduces a hierarchical alignment strategy by hierarchically introducing the model architecture to capture more and richer semantic information at different levels of the Transformer, so as to achieve better semantic alignment between modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flowchart of a hierarchical alignment image-text retrieval method of the present invention;

[0044] Figure 2 is a structural diagram of the Transformer module;

[0045] Figure 3 is a visualization schematic diagram of data dimensionality reduction of image and text features. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The following describes the specific embodiments of the present invention with reference to the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be particularly noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0047] Embodiment

[0048] Figure 1 is a flowchart of a progressive image-text matching method of the present invention.

[0049] In this embodiment, as Figure 1 shown, a hierarchical alignment image-text retrieval method of the present invention can perform image-text retrieval in the way of retrieving text by image or retrieving image by text, and the retrieval principle process is the same. Below, we take retrieving text by image as an example for analysis and description, which specifically includes the following steps:

[0050] S1. Establish a retrieval library;

[0051] Select an image I to be retrieved, and all texts T t containing the text matching the image I are

[0052] S2. Extract image and text features;

[0053] S2.1. Extract image features;

[0054] Compose 12 Transformer modules into the Swin-Transformer model. The 12 Transformer modules are divided into three stages: low, medium, and high. After the pre-training of the Swin-Transformer model is completed, extract the features of the image regions of image I to obtain the feature output maps of the three stages of low, medium, and high, denoted as V low , V middle , V high , thus representing the hierarchical feature V h as:

[0055] V h = {V low , V middle , V high}

[0056]

[0057]

[0058]

[0059] where represents the image features of the k-th, m-th, and g-th image regions divided in the three stages of low, medium, and high. K, M, and G are the number of regions divided in different stages;

[0060] S2.2. Extract text features;

[0061] Compose 12 Transformer modules into the BERT model. The 12 Transformer modules are divided into three stages: low, medium, and high. After the pre-training of the BERT (Bidirectional Encoder Representations from Transformers) model is completed, extract the text features of all texts to obtain the text features T t , denoted as W t low , W t middle , W t high , thus representing the hierarchical feature as:

[0062]

[0063]

[0064]

[0065]

[0066] where \(l = 1, 2, \ldots, L\) denotes the \(l\)-th word feature extracted from the low, medium, and high stages in the \(t\)-th text, and \(L\) is the number of words in the \(t\)-th text;

[0067] In this embodiment, for the extraction of image features, different from the previous traditional convolutional neural network-based models, inspired by the recent rapid development of vision transformers, this embodiment adopts a Transformer-based architecture to learn and extract image features, forming a complementary and unified relationship with the text architecture to obtain a better-compatible modal representation. Specifically, the recently very successful Swin-Transformer model is used as the image backbone, whose performance is significantly better than that of convolutional neural networks and has shown great potential in many vision tasks. It calculates attention through a sliding window method, making the computational complexity linearly related to the image size, and captures multi-scale information by constructing hierarchical feature maps.

[0068] such as Figure 2 As shown, in each Transformer module, the standard multi-head self-attention mechanism is replaced by a sliding window-based self-attention mechanism. Specifically, the standard Transformer implements global self-attention to explore the relationship between a query token and all other tokens, which leads to a computational complexity quadratic with respect to the number of tokens. Due to the high resolution of images, this makes it difficult to be used in many vision applications. To effectively simulate the self-attention process in vision, the Swin-Transformer model divides the image into several non-overlapping windows and only calculates the multi-head self-attention within each window, thus reducing the computational complexity of multi-head self-attention.

[0069] To capture richer modal information, this patent uses the output feature maps of multiple stages in the image and text feature extraction architectures as the hierarchical representations of their respective modalities. Specifically, the Swin-Transformer model is divided into four-stage modules through patch splitting operations. Due to the computational cost brought by a large number of tokens in the first stage, only the 2nd, 3rd, and 4th stages are used as the low, medium, and high-level semantics in this embodiment to achieve multi-level semantic segmentation of images. Similar to the BERT structure, which has the same number of layers as the Swin-Transformer, the outputs of the 4th, 10th, and 12th layers are used to achieve multi-level semantic segmentation of text.

[0070] S3. Hierarchical alignment;

[0071] To achieve finer-grained semantic alignment, the present invention proposes to perform cross-attention operations on semantic modality information at different levels to significantly improve cross-modal retrieval performance.

[0072] S3.1. Calculate the cross-attention weight S of the p-th image region corresponding to the q-th word in the t-th text pq :

[0073]

[0074] where ||·|| represents the L2 norm operation; generically refers to one of the features; generically refers to one of the features; the superscript T represents transpose;

[0075] S3.2. Based on the attention cross-weight, calculate the weight a of the p-th image region corresponding to the q-th word in the t-th text pq ;

[0076]

[0077] S3.3. Sum up all the words in the p-th image region corresponding to the t-th text according to the weights to obtain the weighted sum word feature a of the p-th image region p ;

[0078]

[0079] where is the normalized value of S pq , and λ is the temperature hyperparameter;

[0080] S3.4. Calculate the cosine similarity cos(v p corresponding to the weighted sum word feature a p of the p-th image region p , a p );

[0081] S3.5. Add up the cosine similarities of all the image regions in the same stage among the low, medium, and high stages with the corresponding weighted sum word features to obtain the similarities of each stage, denoted as S low , S middle , S high

[0082]

[0083] where S ∈ S low , S middle , S high ;

[0084] S3.6. Add the similarities of each stage as the retrieval value of image I and the t-th text;

[0085] S4. Obtain the image-text retrieval result;

[0086] Traverse all texts in the retrieval text library, obtain the retrieval values of image I and each text, arrange all the retrieval values from largest to smallest, and then take the text ranked first as the retrieval result.

[0087] In summary, the hierarchical alignment unified Transformer architecture proposed by the present invention can effectively learn the features and common semantic spaces of different modal data, making the final representation more compatible in semantic mapping and similarity measurement, and further improving the accuracy of cross-modal retrieval tasks. To evaluate the performance of the model, the present invention applies the hierarchical unified model to the common data sets MSCOCO and Flickr 30K, where each picture has five description sentences in both data sets. This patent uses the recall rate at K (Recall K) as the evaluation index. The recall rate measures the percentage of true hit rates among the top K rankings in the list. The higher the recall rate, the better the performance. The experimental results are listed in Tables 1 and 2. It can be seen from the experimental results that the method proposed in this patent is superior to all baseline models in both data sets. These results prove the superiority of the hierarchical alignment unified model proposed in this patent in cross-modal retrieval tasks. It can be observed that the performance improvement of text-to-image retrieval is greater than that of image-to-text retrieval. This phenomenon is mainly due to the fact that the overall performance of text-to-image retrieval is not as good as that of image-to-text retrieval, resulting in a smaller value and greater room for improvement.

[0088] Finally, data dimensionality reduction of image and text features is also achieved through the t-SNE (t-distribution - Stochastic Neighbor Embedding) method for visualization, as Figure 3 shown. It can be seen that for a and b using the Transformer architecture for both image and text, similar image-text feature distributions are learned. While c and d with different model architectures show very different distribution patterns. These observations further support the unified structure, enabling the encoder to learn a more compatible representation distribution.

[0089] Table 1. Experimental results on the MSCOCO data set

[0090]

[0091] Table 2. Experimental results on the Flickr 30K data set

[0092]

[0093] Although the above-described illustrative specific embodiments of the present invention have been described to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions made using the concept of the present invention are within the scope of protection.

Claims

1. A hierarchical alignment-based image-text retrieval method, characterized in that It includes the following steps: (1) Establish a retrieval library; Select an image I to be retrieved, and all texts T containing the text matching the image I t are used to form a text set. Finally, the image I and the corresponding text set are used as the retrieval library, where t represents the t-th text; (2) Extract image text features; (2.1) Extract image features; Compose a Swin-Transformer model with 12 Transformer modules. The 12 Transformer modules are divided into three stages: low, medium, and high. After the pre-training of the Swin-Transformer model is completed, perform feature extraction on the image regions of the image I to obtain the feature output maps of the three stages of low, medium, and high, which are respectively denoted as V low , V middle , V high , thus representing the hierarchical feature V h as: V h = {V low , V middle , V high} Among them, represents the image features of the k-th, m-th, and g-th image regions divided in the low, medium, and high stages, where K, M, and G are the number of regions divided in different stages; (2.2) Extract text features; Compose a BERT model with 12 Transformer modules. The 12 Transformer modules are divided into three stages: low, medium, and high. After the pre-training of the BERT model is completed, text feature extraction is performed on all texts to obtain text features T of the low, medium, and high stages t , denoted as W t low ,W t middle ,W t high , thus representing the hierarchical features as: where \(l = 1, 2, \ldots, L\), represents the \(l\)-th word feature extracted from the low, medium, and high stages in the \(t\)-th text, and \(L\) is the number of words in the \(t\)-th text; (3) Hierarchical alignment; (3.1) Calculate the cross-attention weight S of the q-th word in the t-th text corresponding to the p-th image region pq : wherein, ||·|| represents the L2 norm operation; generically refers to one of the features; generically refers to one of the features; the superscript T represents transpose; (3.2) Calculate the weight a of the q-th word in the t-th text corresponding to the p-th image region based on the attention cross weight pq ; (3.3) Sum all the words in the p-th image region corresponding to the t-th text according to their weights to obtain the weighted sum of word features a corresponding to the p-th image region p ; Among them, is the normalized value of S pq , and λ is the temperature hyperparameter; (3.4) Calculate the feature v of the p-th image region p and the corresponding weighted sum of word features a p to obtain the cosine similarity cos(v p , a p ); (3.5) Obtain the similarity of each stage by adding the cosine similarities between all image regions in the same stage among the low, medium, and high stages and the weighted-sum word features, denoted as S low , S middle , S high where S ∈ S low , S middle , S high ; (3.6) Add the similarities at each stage as the retrieval value of image I and the t-th text; (4) Obtain the image text retrieval result; Traverse all texts in the retrieval text library, obtain the retrieval values of image I and each text, arrange all retrieval values from largest to smallest, and then take the text ranked first as the retrieval result.

Images