An image-text retrieval method based on cross-modal semantic alignment

Abstract

The application discloses an image-text retrieval method based on cross-modal semantic alignment. First, the whole image is converted into a group of regional image features, and then the regional image features are processed by a semantic enhancement module to obtain enhanced regional image features. Then, the regional image features related to the word features in the text sentence are updated by two adaptive attention factors. The similarity between each word feature in the text sentence and each regional image feature related to the word feature is calculated, and the similarity score between the whole text sentence and the whole image is calculated. Finally, according to the above steps, when the text sentence or the image is queried, the L pictures or L text sentences with the highest similarity score with the text sentence or the image in the database are retrieved as the retrieval result. The application can more accurately encode the regional image features and significantly promote the word-regional image matching process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image-text retrieval, and particularly relates to an image-text retrieval method based on cross-modal semantic alignment. BACKGROUND

[0002] Image-text retrieval is a basic and challenging task bridging the language and vision fields. Previous works mainly solve the cross-modal image-text retrieval problem from the following directions: 1) coarse-grained retrieval methods directly compute the global similarity between input images and entire texts by mapping two heterogeneous modalities into a common embedding space; 2) fine-grained matching methods automatically align image regions and text segments by exploring their fine-grained cross-modal correspondences.

[0003] The general process of fine-grained image-text matching is as follows: 1) image and text feature representation; 2) design of cross-modal attention module; 3) design of loss function. The most classic method is SCAN (Stacked Cross Attention Network), which infers image-text similarity by discovering all potential alignment relationships between image regions and words in sentences. This method can capture fine-grained interactions between vision and language, making image-text matching more interpretable, and subsequent image-text retrieval models basically continue the main idea of SCAN.

[0004] However, these methods have the following problems: (1) modeling the relationship between object regions and directly performing local-level or global-level matching, in which each region is treated equally, which can impair the accuracy of representation; (2) using one-time forward correlation or aggregation strategies with complex architectures or additional information, while ignoring the regulatory ability of network feedback. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an image-text retrieval method based on cross-modal semantic alignment, which can more accurately encode region-level image features and significantly promote the word-region-level image matching process.

[0006] The technical scheme of the present application is as follows:

[0007] An image-text retrieval method based on cross-modal semantic alignment, specifically comprising the following steps:

[0008] (1) first, the entire image is converted into a set of region-level image features, and then the region-level image features are processed using a semantic enhancement module to obtain enhanced region-level image features;

[0009] (2), two adaptive attention factors are constructed, the two adaptive attention factors are respectively an adaptive channel weight vector for correcting the correlation between the regional image features and the word features in the text sentence, and a weight vector for adjusting the attention distribution, and then the regional image features related to the word features in the text sentence in the regional image features are updated through the two adaptive attention factors;

[0010] (3), the similarity of each word feature in the text sentence and each regional image feature related to the word feature is calculated, and then the similarity score between the entire text sentence and the entire image is calculated;

[0011] (4), according to the above steps (1)-(3), when querying the text sentence, L pictures with the highest similarity score in the database are retrieved as the retrieval result; or when querying the image, L text sentences with the highest similarity score in the database are retrieved as the retrieval result; L is an integer not less than 1.

[0012] The specific steps of first converting the entire image into a set of regional image features, and then processing the regional image features by using the semantic enhancement module to obtain the enhanced regional image features are as follows:

[0013] S11, first convert the entire image I into a set of regional image features Wherein, m i represents a regional image feature of the entire image I, and K represents the number of regional image features in the entire image I;

[0014] S12, processing each regional image feature by using the semantic enhancement module, first input a set of regional image features M, and perform initial global embedding by using a regional feature mapping function local(·) and a global embedding mapping function global(·), as shown in the following formula (1):

[0015]

[0016] In formula (1), W l and W g are learnable parameter matrices;

[0017] S13, the initial global embedding of all regional image features is aggregated by matrix element corresponding multiplication and a softmax function, as shown in the following formula (2), and then the global embedding m g is calculated by the following formula (3):

[0018]

[0019] In formula (2) and formula (3), W denotes the combination of Tanh activation function and batch normalization function, softmax(·) denotes the softmax activation function, denotes the element-wise multiplication of matrices, W a is a learnable parameter matrix, w i is a weight coefficient, represents the relevance between the region-level image feature m i and the initial global embedding, and ‖‖2 denotes the L2 norm, i.e., the Euclidean distance.

[0020] S14, obtaining the region-level image feature enhanced by the whole image by executing the self-attention algorithm Specifically, see formula (4):

[0021]

[0022] In formula (4), x i denotes the region-level image feature m i enhanced by the semantic enhancement.

[0023] The two adaptive attention factors are specifically constructed by the following steps.

[0024] S21, inputting the region-level image feature enhanced by the whole image A set of word features of the text sentence E Wherein, t j denotes the j-th word feature, and N denotes the number of word features in the text sentence.

[0025] S22, for the word feature t j and the region-level image feature m j related to the word feature t The initial value of the region-level image feature enhanced by the semantic enhancement, first construct the alignment vector a j to encode the difference relationship between t j and Specifically, see formula (5):

[0026]

[0027] In formula (5), Norm is the vector normalization operation, and linear is the linear transformation.

[0028] S23, the alignment vector a j is used to learn two adaptive attention factors, the two adaptive attention factors are respectively an adaptive channel weight vector p i used to correct the relevance between the region-level image feature x j and the word feature t j in the text sentence, and a weight vector q jThe calculation formulas are shown in equations (6) and (7) below:

[0029] p j =tanh(W p′ (tanh(W p a j ))) (6);

[0030] q j =[W q′ (tanh(W q a j ))] + (7);

[0031] In equations (6) and (7), W p W p′ W q and W q′ It is a learnable parameter matrix, tanh(·) denotes the tanh activation function, [·] + This indicates taking the larger value of the variable than 0.

[0032] The method of updating the region-level image features related to a word feature in a text sentence by using two adaptive attention factors is specifically shown in the following formula (8):

[0033]

[0034] In equation (8), Representative and word features t j Relevant regional image features; ||| represents the L1 norm, i.e., Manhattan distance; ⊙ represents the XOR operation.

[0035] The specific steps for calculating the similarity between the features of each word in the text sentence and each region-level image feature associated with it, and then calculating the similarity score between the entire text sentence and the entire image, are as follows:

[0036] S31. First, calculate the features t of all words in the text sentence. j and the updated version with t j Related regional image features Each region-level image feature The cosine similarity between them is calculated and then subjected to softmax normalization. The processing procedure is shown in the following equation (9):

[0037]

[0038] In equation (9), The similarity between the i-th region-level image feature and the j-th word feature was measured. + This indicates taking the larger value of the variable than 0;

[0039] S32, the similarity score S between the whole image I and the whole text sentence E is calculated by average pooling AVG (I,E), the calculation process is shown in the following formula (10):

[0040]

[0041] Advantages of the present application:

[0042] (1), the present application utilizes the semantic enhancement module to process the regional level image features, obtains the enhanced regional level image features, and enhances the regional level image features through global representation, compared with directly performing regional level or global level matching, the image visual features can be more accurately encoded.

[0043] (2), the present application updates the regional level image features related to the word features in the text sentence through two adaptive attention factors, explores the supervision ability of the network itself, effectively updates the regional level image features related to the word features, and thus significantly promotes the word-regional level image matching process. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0046] An image-text retrieval method based on cross-modal semantic alignment, specifically comprising the following steps:

[0047] (1), first, the whole image is converted into a set of regional level image features, then the semantic enhancement module is used to process the regional level image features, and the enhanced regional level image features are obtained, and the specific steps are:

[0048] S11, first, the whole image I is converted into a set of regional level image features Wherein, m i represents a regional image feature of the whole image I, K represents the number of regional image features in the whole image I;

[0049] S12, processing each regional-level image feature by using a semantic enhancement module, first inputting a set of regional-level image features M, using a regional-level feature mapping function local(·) and a global embedding mapping function global(·) to perform initial global embedding, see formula (1) below:

[0050]

[0051] In formula (1), W l and W g are learnable parameter matrices;

[0052] S13, aggregating the initial global embedding of all regional-level image features by matrix element corresponding multiplication and softmax function, see formula (2) below, and then calculating the global embedding m g through formula (3) below.

[0053]

[0054] In formula (2) and formula (3), denotes the combination of Tanh activation function and batch normalization function, softmax(·) denotes the softmax activation function, ⊙ denotes matrix element corresponding multiplication, W a is a learnable parameter matrix, w i is a weight coefficient, representing the correlation between the regional-level image feature m i and the initial global embedding, and ‖‖2 denotes the L2 norm, i.e. the Euclidean distance;

[0055] S14, obtaining the enhanced regional-level image features of the whole image by performing a self-attention algorithm See formula (4) below for details:

[0056]

[0057] In formula (4), x i denotes the regional-level image feature m i after semantic enhancement;

[0058] (2), two adaptive attention factors are constructed, and then the regional-level image features related to the word features in the text sentence in the regional-level image features are updated by the two adaptive attention factors, which specifically includes the following steps:

[0059] S21, inputting the enhanced regional-level image features of the whole image a set of word features of the text sentence E Where t j denotes the jth word feature, and N denotes the number of word features in the text sentence.

[0060] S22, Regarding word features t j and word features t j Related regional image features The initial value is the semantically enhanced region-level image feature. First, an alignment vector a is constructed. j Used to encode t j and The differences between them are shown in the following formula (5):

[0061]

[0062] In equation (5), Norm is a vector normalization operation, and linear is a linear transformation;

[0063] S23, Alignment vector a j Two adaptive attention factors are used to learn the region-level image features x. i and word features in text sentences t j The adaptive channel weight vector p of the correlation between them j and the weight vector q used to adjust the attention distribution. j The calculation formulas are shown in equations (6) and (7) below:

[0064] p j =tanh(W p′ (tanh(W p a j ))) (6);

[0065] q j =[W q′ (tanh(W q a j ))] + (7);

[0066] In equations (6) and (7), W p W p′ W q and W q′ It is a learnable parameter matrix, tanh(·) denotes the tanh activation function, [·] + This indicates taking the larger value of the variable than 0;

[0067] S24. Update the region-level image features related to a word feature in the text sentence using two adaptive attention factors, as shown in the following formula (8):

[0068]

[0069] In equation (8), represent the region-level image features related to the word feature t j The ‖‖ represents the L1 norm, i.e., the Manhattan distance; the represents the exclusive or operation;

[0070] (3) Calculate the similarity between each word feature in the text sentence and each region-level image feature related to it, and then calculate the similarity score between the entire text sentence and the entire image. The specific steps are as follows:

[0071] S31, first calculate the cosine similarity between each region-level image feature in the updated region-level image features related to the word feature t j in the text sentence and each word feature t j and perform softmax normalization processing. The processing process is shown in the following formula (9):

[0072]

[0073] In formula (9), measures the similarity between the i-th region-level image feature and the j-th word feature, and [·] + represents the larger value of the variable and 0.

[0074] S32, calculate the similarity score S AVG (I, E) between the entire image I and the entire text sentence E by average pooling. The calculation process is shown in the following formula (10):

[0075]

[0076] (4) According to steps (1)-(3) above, when querying the text sentence, retrieve the L images with the highest similarity score in the database as the retrieval result; or when querying the image, retrieve the L text sentences with the highest similarity score in the database as the retrieval result; L is an integer not less than 1.

[0077] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.​​

Claims

1. An image-text retrieval method based on cross-modal semantic alignment, characterized in that: Specifically comprising the following steps: (1) first, the whole image is converted into a set of region-level image features, and then the region-level image features are processed by a semantic enhancement module to obtain enhanced region-level image features, and the specific steps are as follows: S11, first convert the whole image I into a set of region-level image features where m i denotes a region image feature of the whole image I, and K denotes the number of region image features in the whole image I; S12, processing each region-level image feature by using a semantic enhancement module, first inputting a set of region-level image features M, and using a region-level feature mapping function local(·) and a global embedding mapping function global(·) for initial global embedding, as shown in the following formula (1): In formula (1), W l and W g are learnable parameter matrices; S13. The initial global embeddings of all region-level image features are aggregated by multiplying the corresponding matrix elements and using the softmax function, as shown in equation (2) below. Then, the global embedding m is calculated using equation (3) below. g ; In formula (2) and formula (3), represents the combination of Tanh activation function and batch normalization function, softmax(·) represents the softmax activation function, and represents the corresponding multiplication of matrix elements, W a is a learnable parameter matrix, w i is a weight coefficient, and represents the correlation between the region-level image feature m i and the initial global embedding, and ‖‖2 represents the L2 norm, that is, the Euclidean distance; S14, obtain the region-level image features of the enhanced whole image by performing a self-attention algorithm Specifically, see the following formula (4): In formula (4), x i representing regional-level image features m i the regional-level image features after semantic enhancement (2) two adaptive attention factors are constructed, the two adaptive attention factors are respectively an adaptive channel weight vector for correcting the correlation between the region-level image features and the word features in the text sentence, and a weight vector for adjusting the attention distribution, and then the region-level image features related to the word features in the text sentence in the region-level image features are updated through the two adaptive attention factors; The two adaptive attention factors are constructed, and the specific steps are as follows: S21, input the whole image enhanced region-level image features A set of word features of the text sentence E Wherein, t j Indicates the jth word feature, and N indicates the number of word features in the text sentence. S22, for the word feature t j and the region-level image feature j related to the word feature t The initial value of the region-level image feature j is the semantic enhanced region-level image feature, and an alignment vector a j is first constructed to encode the difference relationship between t and In formula (5), Norm is a vector normalization operation, and linear is a linear transformation. S23, aligning the vector a j for learning two adaptive attention factors, the two adaptive attention factors being respectively an adaptive channel weight vector p i for correcting the relevance between the region-level image feature x j and the word feature t j in the text sentence, and a weight vector q j for adjusting the attention distribution, the calculation formulas being respectively as follows: p j = tanh(W p′ (tanh(W p a j ))) (6); q j = [W q′ (tanh(W q a j )) + (7) In formula (6) and formula (7), W p , W p′ , W q , and W q′ are learnable parameter matrices, tanh(·) represents a tanh activation function, [·] + represents taking the larger value of a variable and 0. (3) the similarity between each word feature in the text sentence and each region-level image feature related to it is calculated, and then the similarity score between the whole text sentence and the whole image is calculated; (4) according to steps (1)-(3) above, when querying the text sentence, the L pictures with the highest similarity score with the text sentence in the database are retrieved as the retrieval result; or when querying the image, the L text sentences with the highest similarity score with the query image in the database are retrieved as the retrieval result; L is an integer not less than 1.

2. The image-text retrieval method based on cross-modal semantic alignment according to claim 1, characterized in that: The region-level image features related to the word features in the text sentence in the region-level image features are updated through the two adaptive attention factors, as shown in the following formula (8): In formula (8), representing a region-level image feature associated with the word feature t j representing a region-level image feature associated with the word feature t ‖‖ represents L1 norm, that is, Manhattan distance; ⊙ represents exclusive or operation.

3. The image-text retrieval method based on cross-modal semantic alignment according to claim 2, characterized in that: The similarity between each word feature in the text sentence and each region-level image feature related to it is calculated, and then the similarity score between the whole text sentence and the whole image is calculated, and the specific steps are as follows: S31、First, calculate all word features t in the text sentence j and the updated region-level image features related to t j The cosine similarity between each region-level image feature in the updated region-level image features is calculated, and softmax normalization is performed, as shown in the following formula (9):​ In formula (9), The similarity between the i-th region-level image feature and the j-th word feature is measured, [·] + denotes the larger value between the variable and 0; S32, calculate the similarity score S between the whole image I and the whole text sentence E by average pooling AVG (I,E), the calculation process is shown in the following formula (10):

Images