A method for image-text matching based on cross-modal mutual attention mechanism

By introducing cross-modal mutual attention mechanism and the ROI features of the object detection network in graphic and text matching, combined with text feature extraction of Bert model, the problem of lack of semantic distance and fine-grained matching of graphic and text similarity calculation in the prior art is solved. , achieving better picture and text matching effect.

CN114492646BActive Publication Date: 2024-11-08BEIJING UNIV OF POSTS & TELECOMM +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210105762.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-28
Publication Date
2024-11-08
Estimated Expiration
2042-01-28

Smart Images

  • Figure CN114492646B_ABST
    Figure CN114492646B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for image-text matching based on a cross-modal mutual attention mechanism, the method comprising the following steps: extracting image semantic features using a target detection network, and extracting text semantic features using a Chinese pre-trained model; calculating the cos similarity between a local image region and a word in a text by a graph-text similarity calculation module, obtaining an attention weight, and calculating a text vector supervised by image attention at the same time; calculating the cos similarity between the text vector supervised by image attention and a local image, and obtaining the similarity between the complete text and the complete image by summing and average pooling; calculating the prediction result of the graph-text similarity. The present invention can infer the similarity between the entire image and the complete sentence, output the similarity value between the image and the sentence, and realize fine-grained graph-text matching from local image to word alignment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of cross-modal image-text matching, and in particular to an image-text matching method based on a cross-modal mutual attention mechanism. Background Art

[0002] Image-text matching is a study of how to measure the semantic similarity between images and text. Image-text matching is closely related to fields such as multimodal alignment and cross-modal retrieval. It is also an interdisciplinary research field between natural language processing and computer vision. Image-text matching is often used in fields such as knowledge retrieval and image-text annotation. In the early stages of research, because the image feature extraction method is completely different from the text feature extraction method, only the "bag of words" of the text and the "bag of visual words" of the image could be used for simple alignment. Later, the field of cross-modal retrieval began to use methods based on canonical correlation analysis (CCA). By linearly mapping different modalities to the same matrix space, the correlation is ranked and compared, and the top ranking is used as the result of image-text matching.

[0003] However, these early image-text matching methods had poor semantic relevance. In recent years, with the development of deep learning, image feature extraction methods have begun to use CNN, ResNet, VGG and other networks to extract features, and Faster-RCNN, YOLO and other networks have also appeared in the field of target detection, which can accurately capture the location of the object interest area in the image, greatly improving the ability to obtain image semantic features.

[0004] With the rise of deep learning to extract semantic features of images and texts, image-text matching has begun to turn to the study of extracting image-text features, mapping them to the same semantic space, and then calculating the similarity of the image-text feature encoding vectors to obtain the image-text matching results. The main image-text similarity measurement methods include: cosine distance, Jaccard distance, Euclidean distance, Hamming distance, etc. However, the current defect of image-text similarity is that the above similarity calculation methods only stay at calculating the distance between encoding vectors, lack the consideration of semantic distance, and lack fine-grained matching alignment.

[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0006] In response to the problems in the related art, the present invention proposes a picture-text matching method based on a cross-modal mutual attention mechanism to overcome the above-mentioned technical problems existing in the existing related technology.

[0007] To this end, the specific technical solution adopted by the present invention is as follows:

[0008] A method for image-text matching based on a cross-modal mutual attention mechanism, the method comprising the following steps:

[0009] S1. Use Faster-RCNN target detection network to extract image semantic features, and use Bert Chinese pre-training model to extract text semantic features;

[0010] S2. Construct a picture-text similarity calculation module, and calculate the cosine similarity between the local image area and the words in the text through the picture-text similarity calculation module to obtain the attention weight, and calculate the text vector supervised by the image attention at the same time;

[0011] S3, calculate the cos similarity between the text vector supervised by image attention and the local image, and obtain the similarity between the complete text and the complete image through summation and average pooling;

[0012] S4. Train and obtain a graph-text matching prediction model, and input the feature encoding of the extracted graph-text data into the graph-text matching prediction model to obtain a prediction result of the graph-text similarity.

[0013] Furthermore, the use of the Faster-RCNN target detection network to extract image semantic features in S1 also includes the following steps:

[0014] The Faster R-CNN target detection network is used to detect the region of interest V in the image. r and POS feature V p Perform extraction;

[0015] V r With V p Splice and get the image semantic feature V s .

[0016] Furthermore, the extraction of text semantic features using the Bert Chinese pre-training model in S1 also includes the following steps:

[0017] Use Chinese word segmentation technology jieba to segment Chinese text to get a word list;

[0018] Use Word Embedding to get the word vector T of the word w , and the word vector T w Input the bidirectional GRU network and extract the text feature T l ;

[0019] Use Bert Chinese pre-trained network to train text features T l Semantic feature T s The semantic feature T s As the final text semantic feature.

[0020] Furthermore, when constructing the image-text similarity calculation module in S2, the image semantic feature V s and text semantic features T s Input the image-text similarity calculation module to obtain the image semantic feature V s and text semantic features T s The similarity between .

[0021] Furthermore, the calculation of the text vector supervised by the image attention in S2 also includes the following steps:

[0022] The image is divided into k local regions according to the number of interest regions k, and one of the local regions is set as V i ;

[0023] Divide the text into n words according to the number of words n, and set one of the words to W j ;

[0024] V i and W j Calculate the cos similarity, the calculation formula is as follows:

[0025] And normalized to get

[0026] Where [x] + ≡max(x,0);

[0027] Calculate the weight a through the softmax function ij , the formula is as follows:

[0028] And the text vector under attention is obtained by weighted summing of words The formula is as follows:

[0029]

[0030] Among the various types, S ij is the similarity between the i-th local image and the j-th word, v i is the i-th local image feature vector, w j is the feature vector of the jth word text, k is the number of interest regions k into which the image is segmented, n is the number of words the sentence text is segmented into, A ij is the weight of the jth word calculated based on its relevance to the image, w j It is the jth word, out of n words.

[0031] Furthermore, obtaining the similarity between the complete text and the complete image by summing and average pooling in S3 further includes the following steps:

[0032] By focusing on the text vector And the cos similarity of the local area of ​​the image Calculate, sum the cos similarities and perform average pooling to obtain the similarity S between the complete text and the complete image AVG (I,T).

[0033] Furthermore, the similarity S between the complete text and the complete image AVG The calculation formula of (I,T) is as follows:

[0034]

[0035]

[0036] In various types, V i is the i-th local image feature, a i is the text feature weighted by image attention supervision, and k is a total of k local image blocks.

[0037] Furthermore, the training of the image-text matching prediction model in S4, and inputting the feature code of the extracted image-text data into the image-text matching prediction model to obtain the prediction result of the image-text similarity further comprises the following steps:

[0038] The random pairing method is used to construct negative examples and train the data. The Triplet Loss function is used as the training target of the image-text matching task, and the image-text matching prediction model is obtained after the training.

[0039] Faster-RCNN and bidirectional GRU are used to extract the feature codes of image and text data respectively, and the extracted feature codes of image and text data are input into the image and text matching prediction model to obtain the prediction results of image and text similarity.

[0040] Furthermore, when the data is trained by constructing negative examples using the random pairing method, the negative example construction strategy further includes the following steps:

[0041] For 40% of the image data, a random sample is selected from the text collection as a negative example;

[0042] For 60% of the image data, texts extracted from the text collection and containing the same entity words as the actual matching texts are selected as negative examples.

[0043] Furthermore, the image-text matching task uses the Triplet Loss loss function as the training target. The Triplet Loss loss function formula is as follows:

[0044]

[0045] Where [x] + ≡max(x,0), S(I,T) is the image-text similarity in S2, is a negative example of text, is a text negative example, and a is a constant.

[0046] The beneficial effects of the present invention are as follows: the fine-grained image-text matching based on the cross-modal mutual attention mechanism proposed in the present invention introduces the ROI features and POS features output by the target detection network in the process of extracting features from the image, which can not only partition the image, but also enhance the semantic features of the image. In the process of extracting text features, the Bert Chinese pre-training model is introduced for self-supervision to further extract semantic information. Finally, this method introduces a cross-modal attention mechanism to align the local interest area of ​​the image and the words in the text, sum and average pooling to obtain the complete image-text similarity, and infer the complete image-text similarity from the local alignment, thereby achieving a better image-text matching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is an implementation flow chart of a graph-text matching method based on a cross-modal mutual attention mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0050] According to an embodiment of the present invention, a method for image-text matching based on a cross-modal mutual attention mechanism is provided. Considering the semantic distance and fine-grained matching, the present invention introduces a cross-modal mutual attention mechanism and target detection technology. The attention involved in image-text matching includes visual attention and text attention. The word vector is calculated after the text is segmented and the ROI (region of interest) feature and POS (POS is the position in the image) feature of the image are extracted by target detection to obtain fine-grained information. The cross-modal mutual attention module is used to calculate the text vector supervised by the image. Finally, the similarity between the text vector supervised by the image and the image feature is calculated to add certain semantic distance information to the image-text matching. At the same time, the fine-grained matching alignment of words to the ROI region of interest is realized. Experiments have proved that the matching results of the image-text matching scheme are good.

[0051] The method of the present invention is based on a cross-modal mutual attention module, and uses the Faster R-CNN target detection algorithm to extract local image features for images, and uses a bidirectional GRU (GRU is a gated recurrent unit) to extract text context features for sentence texts. The local region features in the image and the word features in the text sentence are encoded as input to the cross-modal mutual attention module, and the purpose is to map the words and image regions to a common embedding space and then align the words and image regions to infer the similarity between the entire image and the complete sentence. Finally, the similarity value between the image and the sentence is output, and a fine-grained image-text matching method for local image to word alignment is realized.

[0052] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to the image-text matching method based on the cross-modal mutual attention mechanism of an embodiment of the present invention, the method includes the following steps:

[0053] S1. Use Faster-RCNN target detection network to extract image semantic features, and use Bert Chinese pre-training model to extract text semantic features;

[0054] The use of the Faster-RCNN target detection network to extract image semantic features in S1 also includes the following steps:

[0055] For the features of image data, the Faster R-CNN target detection network is used to detect the ROI (region of interest) features V of the image. r and POS feature V p Perform extraction;

[0056] V r With V p Splice and get the image semantic feature V s .

[0057] The extraction of text semantic features using the Bert Chinese pre-training model in S1 also includes the following steps:

[0058] For the characteristics of Chinese text data, we use the Chinese word segmentation technology Jieba to segment the Chinese text and get a word list;

[0059] Use Word Embedding to get the word vector T of the word w , and the word vector T w Input the bidirectional GRU network and extract the text feature T l ;

[0060] Use Bert Chinese pre-trained network to train text features T l Semantic feature T s The semantic feature T s As the final text semantic feature.

[0061] S2. Construct a picture-text similarity calculation module, and calculate the cosine similarity (cosine similarity) between the local image area and the words in the text through the picture-text similarity calculation module to obtain the attention weight, and calculate the text vector supervised by the image attention at the same time;

[0062] When constructing the image-text similarity calculation module in S2, the image semantic feature V s and text semantic features T s Input the image-text similarity calculation module to obtain the image semantic feature V s and text semantic features T s The similarity between .

[0063] The calculation of the text vector supervised by the image attention in S2 also includes the following steps:

[0064] The image is divided into k local regions according to the number of ROIs k, and one of the local regions is set as V i ;

[0065] Divide the text into n words according to the number of words n, and set one of the words to W j ;

[0066] V i and W j The cos similarity is calculated to measure the importance of the image area to the word. The calculation formula is as follows:

[0067] And normalized to get

[0068] Where [x] + ≡max(x,0);

[0069] Calculate the weight a through the softmax function ij , the formula is as follows:

[0070] And the text vector under attention is obtained by weighted summing of words The formula is as follows:

[0071]

[0072] Among the various types, S ij is the similarity between the i-th local image and the j-th word, v i is the i-th local image feature vector, w j is the feature vector of the jth word text, k is the number of interest regions k into which the image is segmented, n is the number of words the sentence text is segmented into, A ij is the weight of the jth word calculated based on its relevance to the image, w j It is the jth word, out of n words.

[0073] S3, calculate the cos similarity between the text vector supervised by image attention and the local image, and obtain the similarity between the complete text and the complete image through summation and average pooling;

[0074] The process of obtaining the similarity between the complete text and the complete image by summing and averaging pooling in S3 further includes the following steps:

[0075] By focusing on the text vector And the cos similarity of the local area of ​​the image Calculate, sum the cos similarities and perform average pooling to obtain the similarity S between the complete text and the complete image AVG (I,T).

[0076] The similarity S between the complete text and the complete image AVG The calculation formula of (I,T) is as follows:

[0077]

[0078]

[0079] In various types, V i is the i-th local image feature, a i is the text feature weighted by image attention supervision, and k is a total of k local image blocks.

[0080] S4. Train and obtain a graph-text matching prediction model, and input the feature encoding of the extracted graph-text data into the graph-text matching prediction model to obtain a prediction result of the graph-text similarity.

[0081] The training of the image-text matching prediction model in S4, and inputting the feature code of the extracted image-text data into the image-text matching prediction model to obtain the prediction result of the image-text similarity further comprises the following steps:

[0082] The random pairing method is used to construct negative examples and train the data. The Triplet Loss function is usually used as the training target of the image-text matching task. The training goal is to reduce the value of the loss function, and the image-text matching prediction model is obtained after the training.

[0083] Faster-RCNN and bidirectional GRU are used to extract the feature codes of image and text data respectively, and the extracted feature codes of image and text data are input into the image and text matching prediction model to obtain the prediction results of image and text similarity. The results can be used for tasks such as image and text retrieval and image annotation.

[0084] When the data is trained by constructing negative examples using the random pairing method, the negative example construction strategy also includes the following steps:

[0085] For 40% of the image data, a random sample is selected from the text collection as a negative example;

[0086] For 60% of the image data, texts extracted from the text collection and containing the same entity words as the actual matching texts are selected as negative examples.

[0087] The image-text matching task uses the Triplet Loss loss function as the training target. The Triplet Loss loss function formula is as follows:

[0088]

[0089] Where [x] + ≡max(x,0), S(I,T) is the image-text similarity in S2, is a negative example of text, is a negative example of text, and a is a constant. The training goal is to reduce the value of the loss function, and after the training, the image-text matching prediction model is obtained.

[0090] In summary, the fine-grained image-text matching based on the cross-modal mutual attention mechanism proposed in the present invention introduces the ROI features and POS features output by the target detection network in the process of extracting features from the image, which can not only partition the image, but also enhance the semantic features of the image. In the process of extracting text features, the Bert Chinese pre-training model is introduced for self-supervision to further extract semantic information. Finally, this method introduces a cross-modal attention mechanism to align the local interest area of ​​the image and the words in the text, sum and average pooling to obtain the complete image-text similarity, and infer the complete image-text similarity from the local alignment, thereby achieving a better image-text matching effect.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for image-text matching based on cross-modal mutual attention mechanism, It is characterized in that The method comprises the following steps: S1. Use the object detection network to extract image semantic features, and use the Chinese pre-trained model to extract text semantic features; S2. Construct a picture-text similarity calculation module, and calculate the cosine similarity between the local image area and the words in the text through the picture-text similarity calculation module to obtain the attention weight, and calculate the text vector supervised by the image attention at the same time; S3, calculate the cos similarity between the text vector supervised by image attention and the local image, and obtain the similarity between the complete text and the complete image through summation and average pooling; S4. Train and obtain a graph-text matching prediction model, and input the feature encoding of the extracted graph-text data into the graph-text matching prediction model to obtain a prediction result of the graph-text similarity.

2. According to claim 1, a method for image-text matching based on a cross-modal mutual attention mechanism, It is characterized in that The step of extracting image semantic features using a target detection network in S1 further includes the following steps: The target detection network is used to detect the region of interest V in the image. r and POS feature V p Perform extraction; V r With V p Splice and get the image semantic feature V s .

3. According to claim 1, a method for image-text matching based on a cross-modal mutual attention mechanism, It is characterized in that The extraction of text semantic features using the Chinese pre-training model in S1 also includes the following steps: Use Chinese word segmentation technology jieba to segment Chinese text to get a word list; Use Word Embedding to get the word vector T of the word w , and the word vector T w Input the bidirectional GRU network and extract the text feature T l ; Use Bert Chinese pre-trained network to train text features T l Semantic feature T s The semantic feature T s As the final text semantic feature.

4. According to claim 3, a method for image-text matching based on a cross-modal mutual attention mechanism, It is characterized in that When constructing the image-text similarity calculation module in S2, the image semantic feature V s and text semantic features T s Input the image-text similarity calculation module to obtain the image semantic feature V s and text semantic features T s The similarity between .

5. According to claim 1, a method for image-text matching based on a cross-modal mutual attention mechanism, It is characterized in that The calculation of the text vector supervised by the image attention in S2 also includes the following steps: The image is divided into k local regions according to the number of interest regions k, and one of the local regions is set as V i ; Divide the text into n words according to the number of words n, and set one of the words to W j ; V i and W j Calculate the cos similarity, the calculation formula is as follows: And normalized to get Where [x] + ≡max(x,0); Calculate the weight a through the softmax function ij , the formula is as follows: And the text vector under attention is obtained by weighted summing of words The formula is as follows: Among the various types, S ij is the similarity between the i-th local image and the j-th word, v i is the i-th local image feature vector, w j is the feature vector of the jth word text, k is the number of interest regions k into which the image is segmented, n is the number of words the sentence text is segmented into, A ij is the weight of the jth word calculated based on its relevance to the image, w j It is the jth word, out of n words.

6. According to claim 5, a method for image-text matching based on a cross-modal mutual attention mechanism, It is characterized in that The similarity between the complete text and the complete image obtained by summing and average pooling in S3 also includes the following steps: By focusing on the text vector And the cos similarity of the local area of ​​the image Calculate, sum the cos similarities and perform average pooling to obtain the similarity S between the complete text and the complete image AVG (I,T).

7. The image-text matching method based on cross-modal mutual attention mechanism according to claim 6, It is characterized in that The similarity S between the complete text and the complete image AVG The calculation formula of (I,T) is as follows: In various types, V i is the i-th local image feature, a i is the text feature weighted by image attention supervision, and k is a total of k local image blocks.

8. The image-text matching method based on cross-modal mutual attention mechanism according to claim 1, It is characterized in that The training step in S4 to obtain a graph-text matching prediction model, and inputting the feature code of the extracted graph-text data into the graph-text matching prediction model to obtain a prediction result of the graph-text similarity also includes the following steps: The random pairing method is used to construct negative examples and train the data. The Triplet Loss function is used as the training target of the image-text matching task, and the image-text matching prediction model is obtained after the training. Faster-RCNN and bidirectional GRU are used to extract the feature codes of image and text data respectively, and the extracted feature codes of image and text data are input into the image and text matching prediction model to obtain the prediction results of image and text similarity.

9. The image-text matching method based on cross-modal mutual attention mechanism according to claim 8, It is characterized in that When the data is trained by constructing negative examples using the random pairing method, the negative example construction strategy also includes the following steps: For 40% of the image data, a random sample is selected from the text collection as a negative example; For 60% of the image data, texts extracted from the text collection and containing the same entity words as the actual matching texts are selected as negative examples.

10. The image-text matching method based on cross-modal mutual attention mechanism according to claim 9, It is characterized in that The image-text matching task uses the Triplet Loss function as the training target. The Triplet Loss function formula is as follows: Where [x] + ≡max(x,0), S(I,T) is the image-text similarity in S2, is a negative example of text, is a text negative example, and a is a constant.