Fine-grained image and text retrieval method and system based on Transformer model

Abstract

The invention discloses a fine-grained image-text retrieval method based on a Transformer model, comprising: obtaining a region vector group of an image target region and a word vector group of a text; optimizing each target region of the image using a trained self-attention Transformer model so that each target region obtains effective information of other surrounding target regions; determining the meaning information of the current target word according to the semantic information of the text using a trained self-attention Transformer model; performing cross-modal information processing using a trained mutual-attention Transformer model so that the region vector group includes key information and the word vector group includes detail information; respectively calculating the similarity between each region vector and each word vector obtained in the end, obtaining a fine-grained semantic similarity matrix between the input image and the input text, and obtaining a retrieval result. The retrieval performance and accuracy are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image and text retrieval, and specifically relates to a fine-grained image and text retrieval method and system based on a Transformer model. Background Art

[0002] With the development of Internet technology, various applications and web pages generate a large number of pictures and texts every day, and there may be certain connections between these pictures and texts.

[0003] In practical applications, images corresponding to texts can be retrieved based on cross-modal retrieval algorithms. In related technologies, cross-modal retrieval algorithms mainly extract image features of all images in an image library and text features of texts, and determine the similarity between each image and text based on the image features and text features, and then determine the image with the highest similarity to the text from the image library.

[0004] There are three common modes in existing image and text retrieval technology:

[0005] The first category is based on subspace learning, which aims to obtain a common subspace that can measure the similarity of features of different modal data. Due to the high dimensionality of the original data, this type of method often takes a long time to train.

[0006] The second category is the topic model-based method, which explores the association between multimodal data by constructing a "hidden semantic space". Although this type of method can directly associate the semantics of different modal data, it fails to preserve the similarity structure between data samples.

[0007] The third category is the method based on deep learning. The current retrieval method based on deep learning can effectively realize semantic retrieval, but the accuracy of retrieval is not satisfactory. Therefore, the present invention comes. Summary of the invention

[0008] In view of the above-mentioned technical problems, the purpose of the present invention is to provide a fine-grained image-text retrieval method and system based on the Transformer model. Through the self-attention Transformer and the mutual attention Transformer, the correlation features within the modality and between the modalities can be unified for image-text matching calculations, thereby improving the retrieval performance and accuracy.

[0009] The technical solution of the present invention is:

[0010] A fine-grained image and text retrieval method based on a Transformer model includes the following steps:

[0011] S01: Obtain a region vector group of the target region of the image and a word vector group of the text;

[0012] S02: Use the trained self-attention Transformer model to optimize each target area of ​​the image so that each target area obtains valid information from other surrounding target areas; Use the trained self-attention Transformer model to determine the meaning information of the current target word based on the semantic information of the text;

[0013] S03: Use the trained mutual attention Transformer model to process cross-modal information and interact the information between image and text, so that the region vector group includes key information and the word vector group includes detail information;

[0014] S04: Calculate the similarity between each region vector in the final region vector group and each word vector in the word vector group, obtain a fine-grained semantic similarity matrix between the input image and the input text, and obtain the retrieval result.

[0015] In the preferred technical solution, the method for obtaining the regional vector group in step S01 includes:

[0016] S11: Use a detection algorithm to perform target detection on the input image to obtain multiple target areas;

[0017] S12: Use the ROI pooling method to pool the image target area into a corresponding region vector group.

[0018] In the preferred technical solution, the word vector group acquisition method in step S01 includes:

[0019] S13: Use a word segmentation algorithm to divide the input text into several words;

[0020] S14: Use the Bi-GRU encoding method to encode text words into word vector groups.

[0021] In the preferred technical solution, the self-attention Transformer model in step S02 includes at least one layer of Transformer network, and the Transformer network includes a multi-head self-attention module and a feedforward module, and the input and output of each module are cascaded together through a residual connection;

[0022] The multi-head self-attention module maps the input vector group into a query encoding feature vector Q through three mapping networks before calculating the multi-head self-attention of the input feature i , key encoding feature vector K i and the meaning encoding feature vector V i , used as the self-attention vector for subsequent calculation of input features;

[0023] The single-head self-attention is calculated by scaling the dot product attention:

[0024]

[0025] Among them, att i is the i-th head self-attention feature vector, softmax is the activation function, is the scaling constant;

[0026] All single-head self-attention feature vectors are stacked together to calculate the attention similarity matrix Among them, a ij Represents the semantic similarity between the feature vector of the i-th region and the feature vector of the j-th region;

[0027] Normalize the attention similarity matrix through the activation function;

[0028] Map the attention similarity matrix back to the dimension of the input feature, and then concatenate it in a residual way to get the output vector;

[0029] The obtained output vector is updated twice through the feedforward module to obtain an output vector group.

[0030] In the preferred technical solution, the mutual attention Transformer model of step S03 includes at least one layer of Transformer network, the Transformer network includes a multi-head mutual attention module and a feedforward module, and the input and output of each module are cascaded together through a residual connection; the multi-head mutual attention module includes an image multi-head attention module and a word multi-head attention module;

[0031] When the region vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are obtained by mapping the word vector group; when the word vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are obtained by mapping the region vector group;

[0032] The single-head mutual attention of the region vector group and the single-head mutual attention of the word vector group are calculated:

[0033]

[0034]

[0035] in, is the i-th head mutual attention feature vector of the region vector group, att i is the i-th head mutual attention feature vector of the word vector group, are the query, key and meaning encoding feature vectors of the region vector group, Group query, key, and meaning encoding feature vectors for word vectors respectively;

[0036] All single-head mutual attention feature vectors are stacked together to calculate the regional attention similarity matrix and word attention similarity matrix

[0037] The regional attention similarity matrix and the word attention similarity matrix are normalized respectively through the activation function;

[0038] Map the attention similarity matrix back to the dimension of the input feature, and then concatenate it in a residual way to get the output vector;

[0039] The obtained output vector is updated twice through the feed-forward module to obtain the final region vector group and word vector group.

[0040] In a preferred technical solution, the feedforward module includes a nonlinear feedforward network, and the nonlinear feedforward network includes two layers of fully connected networks and one layer of activation layer network.

[0041] In the preferred technical solution, the method for calculating the fine-grained semantic similarity matrix in step S04 includes:

[0042] The cosine similarity calculation function is used to calculate the similarity between each region vector and each word vector. The calculation formula is:

[0043]

[0044] In the formula, Respectively represent the i-th region vector of the final region vector group and the j-th word vector of the final word vector group; s ij is the semantic similarity value between the i-th region vector and the j-th word vector; is the second norm of the i-th region vector, is the bi-norm of the j-th word vector;

[0045] Get the fine-grained semantic similarity matrix

[0046] In the preferred technical solution, after obtaining the fine-grained semantic similarity matrix in step S04, the method further includes:

[0047] Pooling is performed using the LogSumExp pooling method, and finally the semantic similarity value s between the input image and the input text is obtained. The LogSumExp pooling method is:

[0048]

[0049] Where m and n are the number of image target regions and the number of text words, respectively, and τ is the hyperparameter when calculating LogSumExp.

[0050] The present invention also discloses a fine-grained image-text retrieval system based on a Transformer model, comprising:

[0051] An acquisition module is used to acquire a region vector group of a target region of an image and a word vector group of a text;

[0052] The self-attention Transformer model optimization module uses the trained self-attention Transformer model to optimize each target area of ​​the image so that each target area obtains valid information of other surrounding target areas; the trained self-attention Transformer model is used to determine the meaning information of the current target word based on the semantic information of the text;

[0053] The mutual attention Transformer model processing module uses the trained mutual attention Transformer model to perform cross-modal information processing and interact the information between images and texts, so that the region vector group includes key information and the word vector group includes detail information;

[0054] The image-text retrieval module calculates the similarity between each region vector in the final region vector group and each word vector in the word vector group, obtains a fine-grained semantic similarity matrix between the input image and the input text, and obtains the retrieval result.

[0055] In a preferred technical solution, the self-attention Transformer model includes at least one layer of Transformer network, and the Transformer network includes a multi-head self-attention module and a feedforward module, and the input and output of each module are cascaded together through a residual connection;

[0056] The multi-head self-attention module maps the input vector group into a query encoding feature vector Q through three mapping networks before calculating the multi-head self-attention of the input feature i , key encoding feature vector K i and the meaning encoding feature vector V i , used as the self-attention vector for subsequent calculation of input features;

[0057] The single-head self-attention is calculated by scaling the dot product attention:

[0058]

[0059] Among them, att i is the i-th head self-attention feature vector, softmax is the activation function, is the scaling constant;

[0060] All single-head self-attention feature vectors are stacked together to calculate the attention similarity matrix Among them, a ij Represents the semantic similarity between the feature vector of the i-th region and the feature vector of the j-th region;

[0061] Normalize the attention similarity matrix through the activation function;

[0062] Map the attention similarity matrix back to the dimension of the input feature, and then concatenate it in a residual way to get the output vector;

[0063] The obtained output vector is updated twice through the feedforward module to obtain an output vector group.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1) Easy to deploy, and can realize multimodal retrieval end-to-end. Fine-grained image targets and text words are encoded successively through self-attention Transformer and mutual-attention Transformer to realize the retrieval function. This method is based on the Transformer model that has made many breakthroughs in the fields of language and vision, realizes the information interaction between intra-modal data and inter-modal data, improves the performance and accuracy of retrieval, can effectively save labor, reduce labor intensity, improve work efficiency, and has high detection accuracy.

[0066] 2) High retrieval accuracy: The Transformer-based method proposed in this method can unify the correlation features within and between modalities for image-text matching calculation, which can achieve excellent retrieval results. And it has excellent retrieval performance.

[0067] 3) The model can realize semantic retrieval. Since it is trained with large-scale data, it realizes the function of semantic retrieval. Therefore, it can retrieve semantically matching results for images or texts that have not been seen before. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0069] Figure 1 This is a flow chart of the fine-grained image and text retrieval method based on the Transformer model of the present invention;

[0070] Figure 2 This is a schematic diagram of the structure of the self-attention Transforner model in this embodiment;

[0071] Figure 3This is a schematic diagram of the structure of the mutual attention Transforner model of this embodiment;

[0072] Figure 4 It is a principle block diagram of the fine-grained image-text retrieval system based on the Transformer model of the present invention;

[0073] Figure 5 This is a workflow diagram of the fine-grained image and text retrieval system based on the Transformer model in this embodiment. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0075] Example

[0076] like Figure 1 As shown in FIG. 1 , a fine-grained image-text retrieval method based on a Transformer model includes the following steps:

[0077] S01: Obtain a region vector group of the target region of the image and a word vector group of the text;

[0078] S02: Use the trained self-attention Transformer model to optimize each target area of ​​the image so that each target area obtains valid information from other surrounding target areas; Use the trained self-attention Transformer model to determine the meaning information of the current target word based on the semantic information of the text;

[0079] S03: Use the trained mutual attention Transformer model to process cross-modal information and interact the information between image and text, so that the region vector group includes key information and the word vector group includes detail information;

[0080] S04: Calculate the similarity between each region vector in the final region vector group and each word vector in the word vector group, obtain a fine-grained semantic similarity matrix between the input image and the input text, and obtain the retrieval result.

[0081] In a preferred embodiment, the method for obtaining the regional vector group in step S01 includes:

[0082] S11: Use a detection algorithm to perform target detection on the input image to obtain multiple target areas;

[0083] S12: Use the ROI pooling method to pool the image target area into a corresponding region vector group.

[0084] The detection algorithm here is an existing detection algorithm and will not be explained here with examples.

[0085] In a preferred embodiment, the word vector group acquisition method in step S01 includes:

[0086] S13: Use a word segmentation algorithm to divide the input text into several words;

[0087] S14: Use the Bi-GRU encoding method to encode text words into word vector groups.

[0088] The word segmentation algorithm here is the existing word segmentation algorithm, and no example is given here.

[0089] A preferred embodiment, such as Figure 2 As shown, the self-attention Transformer model in step S02 includes at least one layer of Transformer network, and the Transformer network includes a multi-head self-attention module and a feedforward module, and the input and output of each module are cascaded together through a residual connection. Figure 2 The self-attention Transformer model consists of 4 stacked Transformer networks.

[0090] Before calculating the multi-head self-attention of the input features, the multi-head self-attention module maps the input vector group into the query encoding feature vector Q through three mapping networks i , key encoding feature vector K i and the meaning encoding feature vector V i , used as the self-attention vector for subsequent calculation of input features;

[0091] The single-head self-attention is calculated by scaling the dot product attention:

[0092]

[0093] Among them, att i is the i-th head self-attention feature vector, softmax is the activation function, is the scaling constant;

[0094] All single-head self-attention feature vectors are stacked together to calculate the attention similarity matrix Among them, a ij Represents the semantic similarity between the feature vector of the i-th region and the feature vector of the j-th region;

[0095] Normalize the attention similarity matrix through the activation function;

[0096] Map the attention similarity matrix back to the dimension of the input feature, and then concatenate it in a residual way to get the output vector;

[0097] The obtained output vector is updated twice through the feedforward module to obtain an output vector group.

[0098] A preferred embodiment, such as Figure 3 As shown, the mutual attention Transformer model of step S03 includes at least one layer of Transformer network, wherein the Transformer network includes a multi-head mutual attention module and a feedforward module, and the input and output of each module are cascaded together through a residual connection; the multi-head mutual attention module includes an image multi-head attention module and a word multi-head attention module; Figure 3 The mutual attention Transformer model consists of 4 stacked Transformer networks.

[0099] When the region vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are obtained by mapping the word vector group; when the word vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are obtained by mapping the region vector group;

[0100] The single-head mutual attention of the region vector group and the single-head mutual attention of the word vector group are calculated:

[0101]

[0102]

[0103] in, is the i-th head mutual attention feature vector of the region vector group, att i is the i-th head mutual attention feature vector of the word vector group, are the query, key and meaning encoding feature vectors of the region vector group, Group query, key, and meaning encoding feature vectors for word vectors respectively;

[0104] All single-head mutual attention feature vectors are stacked together to calculate the regional attention similarity matrix and word attention similarity matrix

[0105] The regional attention similarity matrix and the word attention similarity matrix are normalized respectively through the activation function;

[0106] Map the attention similarity matrix back to the dimension of the input feature, and then concatenate it in a residual way to get the output vector;

[0107] The obtained output vector is updated twice through the feed-forward module to obtain the final region vector group and word vector group.

[0108] In a preferred embodiment, the feedforward module includes a nonlinear feedforward network, and the nonlinear feedforward network includes two layers of fully connected networks and one layer of activation layer network.

[0109] In a preferred embodiment, the method for calculating the fine-grained semantic similarity matrix in step S04 includes:

[0110] The cosine similarity calculation function is used to calculate the similarity between each region vector and each word vector. The calculation formula is:

[0111]

[0112] In the formula, Respectively represent the i-th region vector of the final region vector group and the j-th word vector of the final word vector group; s ij is the semantic similarity value between the i-th region vector and the j-th word vector; is the second norm of the i-th region vector, is the bi-norm of the j-th word vector;

[0113] Get the fine-grained semantic similarity matrix

[0114] In a preferred embodiment, after obtaining the fine-grained semantic similarity matrix in step S04, the method further includes:

[0115] Pooling is performed using the LogSumExp pooling method, and finally the semantic similarity value s between the input image and the input text is obtained. The LogSumExp pooling method is:

[0116]

[0117] Where m and n are the number of image target regions and the number of text words, respectively, and τ is the hyperparameter when calculating LogSumExp.

[0118] like Figure 4 As shown, the present invention also discloses a fine-grained image and text retrieval system based on a Transformer model, comprising:

[0119] An acquisition module 10 acquires a region vector group of a target region of an image and a word vector group of a text;

[0120] The self-attention Transformer model optimization module 20 uses the trained self-attention Transformer model to optimize each target area of ​​the image so that each target area obtains effective information of other surrounding target areas; the trained self-attention Transformer model is used to determine the meaning information of the current target word according to the semantic information of the text;

[0121] The mutual attention Transformer model processing module 30 uses the trained mutual attention Transformer model to perform cross-modal information processing and interact the information between the image and the text, so that the region vector group includes key information and the word vector group includes detail information;

[0122] The image-text retrieval module 40 respectively calculates the similarity between each region vector in the final region vector group and each word vector in the word vector group, obtains a fine-grained semantic similarity matrix between the input image and the input text, and obtains the retrieval result.

[0123] The following is a detailed description of the workflow of the fine-grained image-text retrieval system based on the Transformer model using the best example. Figure 5 As shown, the following steps are included:

[0124] Step 1: Use the detection algorithm to detect the target of the input image and obtain multiple target regions (region proposal), and use the word segmentation algorithm to divide the input text description into several words;

[0125] Step 2: Use the ROI pooling method to pool the image objects into corresponding regional vector groups, and use the Bi-GRU encoding method to encode the text words into word vector groups;

[0126] Step 3: Optimize the region vector group and word vector group obtained in step 2 through the self-attention Transformer model: Based on the image self-attention Transformer model, optimize each target region (region proposal) of the image to more accurately express the information of the current target region; based on the text self-attention Transformer model, clarify the meaning of the current target word according to the overall semantic information of the entire text, and express the information of the current word more clearly;

[0127] like Figure 2 As shown in Figure 1, the Transformer network consists of two modules: a multi-head self-attention module and a feedforward module. The input and output of each module are cascaded together through residual connections. The input region vector group is represented as The word vector group is represented as Among them, m and n are the number of image target areas and the number of text words respectively, and dv ,d t for each region vector dimension and each word dimension respectively.

[0128] Before calculating the multi-head self-attention of the input features, a mapping network is needed to map the input features to a specific dimension. The mapping network is implemented using a fully connected layer. Taking the input region vector group v as an example, three different fully connected networks are used to map the input region vector group v into three encoding feature vectors: query Q, key K, and meaning V, which are used as the self-attention vectors for the subsequent calculation of the input features, as shown in the following formula (1):

[0129]

[0130] In the formula, Q i , K i , V i Encode feature vectors for query Q, key K and meaning V, is the weight matrix, where d v is the dimension of the input feature vector, d is the dimension of the encoded feature vector, and in the present invention, d v is 1024, and d is 512. After calculating the Q, K, and V matrices, the single-head self-attention is calculated by scaling the dot product attention, as shown in the following formula (2):

[0131]

[0132] Among them, att i That is, the self-attention feature vector calculated by the i-th head; scaling constant To ensure Q i K i T The matrix value of will not be too large, thus preventing the attention model training from failing.

[0133] By stacking all the single-head attentions together, we can get the self-attention similarity matrix, as shown in the following formula (3):

[0134] A=Concat(att1,…,att h ) (3)

[0135] Calculated is the attention similarity matrix, where a ij represents the semantic similarity between the feature vector of the i-th target region and the feature vector of the j-th target region; att iIt is the single-head attention defined in formula (2), where the subscript i ranges from 1 to h, and h is the number of attention heads. Concat means vector stacking, that is, multiple attention heads are concatenated together according to the dimension of the vector length to the target area to achieve feature fusion of multiple attention heads.

[0136] The softmax in formula (2) is used as an activation function to normalize the self-attention similarity matrix A, focusing on items with relatively large semantic similarity and selectively ignoring items with small semantic similarity, and updating a ij Get a ij ',Right now

[0137] The self-attention similarity matrix A is mapped back to the dimension of the input feature v, and then concatenated in a residual manner, as shown in the following formula (4):

[0138] v s =v+AW o (4)

[0139] Among them, v and v s are the input and output of the self-attention module respectively; is a mapping matrix. By mapping, the dimension of the obtained multi-head attention is kept consistent with the dimension of the original input region vector group v, and then a residual connection with v is performed.

[0140] Get the updated v through the self-attention module s After that, the feedforward module uses a nonlinear feedforward network to s For secondary update, the nonlinear feedforward network is implemented by two layers of fully connected networks and one layer of activation layer network, as shown in the following formula (5):

[0141]

[0142] In the formula, b1∈R d ; are the weight parameters of the two fully connected networks; f act is an activation layer network. In the present invention, act Defined as the common ReLU activation function, that is, f act =max(0,x); output That is, the final output vector group obtained by the region vector group through a single-layer self-attention Transformer, That is, the dimension of the output vector group is consistent with the dimension of the original input regional vector group, so the regional vector group can be gradually optimized by stacking several layers of self-attention Transformer.

[0143] Several layers of stacked self-attention transformers can enable each target area of ​​the input image to obtain effective information from other surrounding target areas, thereby more accurately expressing the information of the current target. In order to balance the retrieval performance and retrieval efficiency of the present invention, the number of layers of the self-attention transformer should not be too many or too few. In the present invention, the number of layers of the self-attention transformer is heuristically taken as 4. Similarly, for the input word vector group Use the self-attention Transformer described above to encode it and obtain the word vector group optimized by self-attention It can also express the meaning of the target word more clearly.

[0144] Step 4: Cross-modal (image-text) information processing: By building a cross-modal mutual attention Transformer, not only can the corresponding image details be inferred from the text, but it can also select corresponding information from the visual information to assist in matching the corresponding words. This step helps to establish the connection between the visual modality and the language modality.

[0145] The basic structure of the mutual attention Transformer model is the same as that of the self-attention Transformer model, which includes two modules: the multi-head attention module and the feedforward module. Figure 3 shown.

[0146] The difference is that when the multi-head attention module of the mutual attention Transformer obtains the attention similarity matrix, Q, K, and V do not come from the input of the same modality. Specifically, it is shown in the following formula (6):

[0147]

[0148] The parameters in the above formula are basically the same as those in formula (1). The encoded feature vectors representing the query, key, and meaning of the region vector group, The encoded feature vectors representing the query, key, and meaning of the word vector group. Modifying formula (2) yields the mathematical expression of the single-head mutual attention of the region vector group and the single-head mutual attention of the word vector group, as shown in formula (7):

[0149]

[0150] It is worth noting that the biggest difference between formula (7) and formula (2) is that when the region vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are not mapped by the region vector group but by the word vector group; similarly, when the word vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are mapped by the region vector group rather than by itself, which realizes the information transfer between the input image and the text.

[0151] Subsequently, according to formulas (3)-(5), the feature expressions of the region vector group and the word vector group optimized by mutual attention are obtained, as shown in formula (8):

[0152]

[0153] In the formula, v c and They are the regional vector group optimized by the self-attention Transformer model, the regional vector group output by the mutual attention module in the mutual attention Transformer model, and the regional vector group output by the mutual attention Transformer model; t c and For v c and The corresponding word vector group; are the region and word mutual attention similarity matrices respectively; Concat, att, W and b have the same meanings as in equations (3)-(5) and will not be repeated here.

[0154] The number of layers of the mutual attention transformer in the present invention is 4. The mutual attention transformer model realizes the information interaction between images and texts, which can make the target area vector group with redundant information have key information and make the word vector group with more concise information rich in detail information.

[0155] Step 5: As shown in formula (8), the region vector group obtained by optimizing the mutual attention Transformer model Each region vector and word vector group After that, calculate Each region vector in and Each word vector in The fine-grained semantic similarity matrix between the input image and the input text is obtained by calculating the similarity Then perform LogSumExp pooling on the semantic similarity matrix S to obtain the final semantic similarity value s.

[0156] Specifically, the similarity between each region vector and each word vector is calculated by the cosine similarity calculation function to obtain each item in the fine-grained semantic similarity matrix, as shown in the following formula (9):

[0157]

[0158] In the formula, Respectively represent the i-th region vector of the final region vector group and the j-th word vector of the final word vector group; s ij is the semantic similarity value between the i-th region vector and the j-th word vector; is the second norm of the i-th region vector.

[0159] In the obtained area vector group With word vector group The semantic similarity matrix S∈R m×n Finally, the LogSumExp pooling method is used to pool the input image and the input text, and the semantic similarity value between the input image and the input text is obtained. The LogSumExp pooling method is defined as:

[0160]

[0161] Wherein S represents the semantic similarity matrix between the region vector group in the input image and the word vector group in the input text, s represents the final semantic similarity value between the input image and the input text, τ is the hyperparameter when calculating LogSumExp, and in this embodiment, τ is 0.1.

[0162] After calculating the semantic numerical value of a single image and a single paragraph of text, the parameters of the entire model can be optimized by constructing a learning paradigm, and finally the optimal model can be obtained. Through training, the semantic similarity value of semantically matching images and texts is attempted to be increased, and the semantic similarity value of semantically mismatched images and texts is attempted to be reduced. Finally, the trained model can realize cross-modal semantic retrieval and has excellent retrieval performance. In the test phase of image-text retrieval, by calculating the semantic similarity between the input image / text and all samples in the current text / image sample library, the most semantically similar text segment can be returned to obtain the retrieval result, thereby realizing the image-text retrieval function.

[0163] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims

1. A fine-grained image and text retrieval method based on the Transformer model, characterized in that: The following steps are involved: S01: Obtain a region vector group of the target region of the image and a word vector group of the text; S02: using the trained self-attention Transformer model to optimize each target area of ​​the image so that each target area obtains effective information of other surrounding target areas; using the trained self-attention Transformer model to determine the meaning information of the current target word according to the semantic information of the text; the self-attention Transformer model includes at least one layer of Transformer network, the Transformer network includes a multi-head self-attention module and a feedforward module, and the input and output of each module are cascaded together through a residual connection; The multi-head self-attention module maps the input vector group into a query encoding feature vector Q through three mapping networks before calculating the multi-head self-attention of the input feature i , key encoding feature vector K i and the meaning encoding feature vector V i , used as the self-attention vector for subsequent calculation of input features; The single-head self-attention is calculated by scaling the dot product attention: Among them, att i is the i-th head self-attention feature vector, softmax is the activation function, is the scaling constant; All single-head self-attention feature vectors are stacked together to calculate the attention similarity matrix Among them, a ij Represents the semantic similarity between the feature vector of the i-th region and the feature vector of the j-th region; Normalize the attention similarity matrix through the activation function; Map the attention similarity matrix back to the dimension of the input feature, and then concatenate it in a residual way to get the output vector; The obtained output vector is updated twice through the feedforward module to obtain an output vector group; S03: Use the trained mutual attention Transformer model to perform cross-modal information processing, interact the information between the image and the text, so that the region vector group includes key information and the word vector group includes detail information; the mutual attention Transformer model includes at least one layer of Transformer network, the Transformer network includes a multi-head mutual attention module and a feedforward module, and the input and output of each module are cascaded together through residual connections; the multi-head mutual attention module includes an image multi-head attention module and a word multi-head attention module; When the region vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are obtained by mapping the word vector group; when the word vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are obtained by mapping the region vector group; The single-head mutual attention of the region vector group and the single-head mutual attention of the word vector group are calculated: in, is the i-th head mutual attention feature vector of the region vector group, att i is the i-th head mutual attention feature vector of the word vector group, are the query, key and meaning encoding feature vectors of the region vector group, Group query, key, and meaning encoding feature vectors for word vectors, respectively; All single-head mutual attention feature vectors are stacked together to calculate the regional attention similarity matrix and word attention similarity matrix The regional attention similarity matrix and the word attention similarity matrix are normalized respectively through the activation function; Map the attention similarity matrix back to the dimension of the input feature, and then concatenate it in a residual way to get the output vector; The obtained output vector is updated twice through the feed-forward module to obtain the final region vector group and word vector group; S04: Calculate the similarity between each region vector in the final region vector group and each word vector in the word vector group, obtain a fine-grained semantic similarity matrix between the input image and the input text, and obtain the retrieval result.

2. According to the Transformer model-based fine-grained image and text retrieval method of claim 1, it is characterized in that: The method for obtaining the regional vector group in step S01 includes: S11: Use a detection algorithm to perform target detection on the input image to obtain multiple target areas; S12: Use the ROI pooling method to pool the image target area into a corresponding region vector group.

3. The fine-grained image-text retrieval method based on the Transformer model according to claim 1 is characterized in that: The word vector group acquisition method in step S01 includes: S13: Use a word segmentation algorithm to divide the input text into several words; S14: Use the Bi-GRU encoding method to encode text words into word vector groups.

4. The fine-grained image-text retrieval method based on the Transformer model according to claim 1 is characterized in that: The feedforward module includes a nonlinear feedforward network, and the nonlinear feedforward network includes two layers of fully connected networks and one layer of activation layer network.

5. The fine-grained image-text retrieval method based on the Transformer model according to claim 1 is characterized in that: The method for calculating the fine-grained semantic similarity matrix in step S04 includes: The cosine similarity calculation function is used to calculate the similarity between each region vector and each word vector. The calculation formula is: In the formula, Respectively represent the i-th region vector of the final region vector group and the j-th word vector of the final word vector group; s ij is the semantic similarity value between the i-th region vector and the j-th word vector; is the second norm of the i-th region vector, is the bi-norm of the j-th word vector; Get the fine-grained semantic similarity matrix 6. The fine-grained image-text retrieval method based on the Transformer model according to claim 1 or 5, characterized in that: After obtaining the fine-grained semantic similarity matrix in step S04, the method further includes: Pooling is performed using the LogSumExp pooling method, and finally the semantic similarity value s between the input image and the input text is obtained. The LogSumExp pooling method is: Where m and n are the number of image target regions and the number of text words, respectively, and τ is the hyperparameter when calculating LogSumExp.

7. A fine-grained image-text retrieval system based on the Transformer model, characterized in that: include: An acquisition module is used to acquire a region vector group of a target region of an image and a word vector group of a text; A self-attention Transformer model optimization module, which uses a trained self-attention Transformer model to optimize each target area of ​​the image so that each target area obtains effective information of other surrounding target areas; uses the trained self-attention Transformer model to determine the meaning information of the current target word according to the semantic information of the text; the self-attention Transformer model includes at least one layer of Transformer network, and the Transformer network includes a multi-head self-attention module and a feedforward module, and the input and output of each module are cascaded together through a residual connection; The multi-head self-attention module maps the input vector group into a query encoding feature vector Q through three mapping networks before calculating the multi-head self-attention of the input feature i , key encoding feature vector K i and the meaning encoding feature vector V i , used as the self-attention vector for subsequent calculation of input features; The single-head self-attention is calculated by scaling the dot product attention: Among them, att i is the i-th head self-attention feature vector, softmax is the activation function, is the scaling constant; All single-head self-attention feature vectors are stacked together to calculate the attention similarity matrix Among them, a ij Represents the semantic similarity between the feature vector of the i-th region and the feature vector of the j-th region; Normalize the attention similarity matrix through the activation function; Map the attention similarity matrix back to the dimension of the input feature, and then concatenate it in a residual way to get the output vector; The obtained output vector is updated twice through the feedforward module to obtain an output vector group; A mutual attention Transformer model processing module uses a trained mutual attention Transformer model to perform cross-modal information processing, interacts the information between the image and the text, so that the region vector group includes key information and the word vector group includes detail information; the mutual attention Transformer model includes at least one layer of Transformer network, the Transformer network includes a multi-head mutual attention module and a feedforward module, and the input and output of each module are cascaded together through a residual connection; the multi-head mutual attention module includes an image multi-head attention module and a word multi-head attention module; When the region vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are obtained by mapping the word vector group; when the word vector group is used to calculate the single-head mutual attention, its key coding feature vector and meaning coding feature vector are obtained by mapping the region vector group; The single-head mutual attention of the region vector group and the single-head mutual attention of the word vector group are calculated: in, is the i-th head mutual attention feature vector of the region vector group, att i is the i-th head mutual attention feature vector of the word vector group, are the query, key and meaning encoding feature vectors of the region vector group, Group query, key, and meaning encoding feature vectors for word vectors, respectively; All single-head mutual attention feature vectors are stacked together to calculate the regional attention similarity matrix and word attention similarity matrix The regional attention similarity matrix and the word attention similarity matrix are normalized respectively through the activation function; Map the attention similarity matrix back to the dimension of the input feature, and then concatenate it in a residual way to get the output vector; The obtained output vector is updated twice through the feed-forward module to obtain the final region vector group and word vector group; The image-text retrieval module calculates the similarity between each region vector in the final region vector group and each word vector in the word vector group, obtains a fine-grained semantic similarity matrix between the input image and the input text, and obtains the retrieval result.

Images