Image-text retrieval method based on comparative learning and modal fusion

Abstract

The invention discloses an image-text retrieval method based on comparative learning and modal fusion. The method comprises the following steps: giving an original image-text pair; the method comprises the following steps of: 1, respectively carrying out local coding and global coding on an input image based on a target detection model and a deep residual network, and coding an input text based on a pre-training language model; 2, generating more matched image-text pairs (positive sample pairs) and mismatched image-text pairs (negative sample pairs) by adopting a diversified data augmentation strategy; 3, establishing cross-modal hierarchical semantic fusion of the image and the text, and bridging a semantic gap between a visual mode and a language mode; 4, measuring the semantic similarity degree between the image and the text; and 5, based on positive and negative sample pairs generated by data augmentation, carrying out model training through comparative learning. The method solves the problem that an existing image-text retrieval method is insensitive to fine granularity difference between modes.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and natural language processing, and specifically relates to a picture-text retrieval method based on contrastive learning and modal fusion. Background Art

[0002] With the continuous innovation of information technology and the diversification of application needs, multimodal data in various forms such as text, images and videos has increased dramatically, and cross-modal retrieval is gradually becoming a key research and application direction. No longer limited to single modal perception, cross-modal retrieval is committed to integrating information from different modalities such as text, images, and voice to more comprehensively understand and meet user retrieval needs. As one of the mainstream tasks of cross-modal retrieval, image-text retrieval aims to use data in one modality to query data in another modality that is semantically related to it. It usually includes two subtasks: "image-text" retrieval and "text-image" retrieval.

[0003] The core challenge of the image-text retrieval task is to bridge the semantic gap between the visual modality and the linguistic modality, and to achieve unified representation and accurate semantic alignment between heterogeneous modalities (images composed of low-level pixels and texts represented by high-level semantic vectors). Existing image-text retrieval methods can be roughly divided into two categories: (1) Global association: Early image-text retrieval methods belong to global association, that is, the entire image and the entire text are mapped to a potential common subspace through the corresponding deep learning network. The cross-modal semantic association similarity of image and text in this common subspace can be directly measured, and the semantic similarity of matching image-text pairs is higher than that of non-matching image-text pairs. However, early global association methods ignored the fine-grained semantic alignment between image salient region information and text words, which limited the further improvement of the accuracy of image-text semantic association. (2) Local association: In order to solve the limitation of coarse-grained semantic alignment of global association methods, existing image-text retrieval methods aim to further explore the fine-grained semantic alignment relationship between all image regions and text fragments. For example, the cross-attention network SCAN explores all possible local correspondences by introducing a mutual attention mechanism between regions and words, but the current method still has the problem of being insensitive to fine-grained differences between modalities. Summary of the invention

[0004] Purpose of the invention: The present invention proposes a method for image-text retrieval based on contrastive learning and modality fusion, which solves the problem that existing image-text retrieval methods are insensitive to fine-grained differences between modalities.

[0005] Technical solution: To achieve the above-mentioned invention object, the present invention proposes a picture-text retrieval method based on contrastive learning and modality fusion, comprising the following steps:

[0006] (1) For the input image-text pair, use the object detection model to encode the image to obtain the salient region features of each image; use the deep residual network to encode the image to obtain the global features of each image; use the pre-trained language model to encode the word sequence corresponding to the text to obtain the word vector of each word;

[0007] (2) For an input image-text pair, a first data augmentation strategy of changing the text at the original input layer is used to generate a positive sample pair, wherein the positive sample pair is an image-text pair with semantic matching, and a second data augmentation strategy of adding noise at the word sequence vector layer is used to generate a negative sample pair, wherein the negative sample pair is an image-text pair with semantic mismatch;

[0008] (3) Perform cross-modal hierarchical semantic fusion of images and texts to eliminate the semantic gap between visual modalities and language modalities: Use a cross-modal attention mechanism to mine the alignment relationship between all image salient regions and text words to achieve cross-modal local semantic alignment of images and texts; Use a learnable embedding matrix to map the entire image and text to a common subspace to achieve cross-modal global semantic alignment of images and texts;

[0009] (4) The similarity of an image-text pair is determined by both local semantic similarity and global semantic similarity. The local semantic similarity is learned by taking fine-grained image salient regions and text words as objects, and the global semantic similarity is learned by taking the entire image and text as objects.

[0010] (5) Combining the positive sample pairs and negative sample pairs generated in step (2), using a contrastive learning loss function, constraining the similarity of semantically matching image-text pairs to be greater than the similarity of other semantically mismatched image-text pairs, and continuously optimizing the training model.

[0011] Furthermore, in the step (1), the image salient area features are obtained by mapping through the target detection model Faster-RCNN, the image global features are obtained by mapping through the deep residual network ResNet152, and the word vector is obtained by mapping through the pre-trained language model BERT.

[0012] Furthermore, in step (2), the first data augmentation strategy is text splicing and text cropping; the second data augmentation strategy is adding Gaussian noise, disordering, truncation and random discarding of semantic features.

[0013] Furthermore, in step (3), a scaled dot product attention mechanism is used to achieve local semantic alignment and cross-modal hierarchical semantic fusion of image salient areas and text words, and then obtain new global features and local features of the image and text.

[0014] Furthermore, in step (4), a dot product similarity calculation method is used to obtain global semantic similarity and local semantic similarity.

[0015] Furthermore, in step (5), the positive sample pairs and negative sample pairs constructed in step (2) are combined and the contrastive learning loss function InfoNCE is used to perform loss calculation to shorten the distance between the positive sample pairs and increase the distance between the negative sample pairs in the semantic space.

[0016] Beneficial effects: The present invention can effectively improve the accuracy of image-text retrieval and solve the problem that the image-text retrieval algorithm is insensitive to the fine-grained differences between modalities. According to the fine-grained differences between image-text modalities, different data augmentation strategies are used to construct more semantically matched and mismatched image-text pairs, and a method for image-text retrieval based on contrastive learning and hierarchical modal fusion is proposed to improve the accuracy of the model for image-text retrieval. First, the present invention uses the target detection model Faster-RCNN and the deep residual network ResNet101 to encode the salient areas and the overall image of the input image, respectively, and by using the pre-trained language model BERT to encode the word sequence corresponding to the input sentence, the underlying visual encoding ability and semantic encoding ability of the model are enhanced; second, in view of the fine-grained differences between modalities, different data augmentation strategies are used to generate positive and negative samples, increase the number and diversity of training samples, and provide more matching image-text pairs and mismatched image-text for subsequent contrastive learning training, thereby improving the model's image and text. The ability to unify modal representation; third, comprehensively consider the global semantic alignment and local semantic alignment of image and text modalities, which helps to achieve more fine-grained modal fusion; fourth, comprehensively consider the global semantic similarity and local semantic similarity of image and text, which helps to more accurately calculate the matching degree between image-text pairs; fifth, use a contrastive learning algorithm based on image-text similarity to make the representations of matching image-text pairs as similar as possible in the feature space, and the representations of mismatching image-text pairs as different as possible, so as to achieve unified representation and modality alignment of different modalities, and further improve the accuracy of the model for image-text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall framework diagram of the method of the present invention;

[0018] Figure 2 This is an example diagram of the image-text similarity comparison learning algorithm in the method of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further explained below in conjunction with the accompanying drawings. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art fall within the scope of the claims attached to this application.

[0020] This paper proposes an image-text retrieval method based on contrastive learning and modality fusion, which solves the problem that the image-text retrieval algorithm is insensitive to the fine-grained differences between modalities. Figure 1 As shown, the complete process of the present invention includes five parts: image and text feature construction stage, data augmentation stage, modality fusion stage, image and text similarity calculation stage, and contrastive learning training stage. The specific implementation method is described as follows:

[0021] The image and text feature construction stage corresponds to step (1) of the technical solution. The specific implementation method is as follows: for a given input image I, in order to obtain an accurate underlying visual representation, the present invention uses the target detection model Faster-RCNN to extract K significant area features in image I. (K is the number of salient regions, D v is the feature dimension of F), the vector F is mapped to the D-dimensional space through the fully connected layer and normalized to V = {v 1 ,v 2 ,…,v K}∈R K*D (D is the feature dimension of V) as the local area feature of the image. In addition, the deep residual network ResNet152 is used to obtain the global features of the input image I and map it to the D-dimensional space G∈R through the fully connected layer D For a given input text T, in order to obtain accurate high-level semantic representation, the present invention uses the pre-trained language model BERT to extract the word sequence feature W corresponding to the text T = {w 1 ,w 2 ,…,w L}∈R L*D (L is the number of words in the text).

[0022] The data augmentation stage corresponds to step (2) of the technical solution. The specific implementation method is as follows: for the same scene, images convey visual information based on pixels, including color, shape, texture, and spatial relationship. Text, on the other hand, conveys contextual information through words and phrases, and usually cannot convey the same level of fine-grained details as images. Therefore, different data augmentation strategies are used to generate positive and negative samples based on the fine-grained differences between image and text modalities.

[0023] The first data augmentation strategy generates positive samples: a) splicing text: enriching the input text information without introducing erroneous information; b) cropping text: streamlining the input text information.

[0024] The second data augmentation strategy generates negative samples: a) Adding Gaussian noise: text word sequence feature W = {w 1 ,w 2 ,…,w L}∈R L*D Introduce noise that conforms to the normal distribution curve; b) Shuffle: Shuffle the text word sequence features W = {w 1 ,w 2 ,…,w L}∈R L*D c) Cutoff: The text word sequence feature W = {w 1 ,w 2 ,…,w L}∈R L*D Set a row or column in the text to 0; d) Randomly discard semantic features (Dropout): randomly discard text word sequence features W = {w 1 ,w 2 ,…,w L}∈R L*D The elements in .

[0025] The newly generated text and its corresponding image constitute more semantically matched image-text pairs and semantically mismatched image-text pairs for use in the subsequent step (5) of comparative learning training.

[0026] The modal fusion stage corresponds to step (3) of the technical solution. The specific implementation method is as follows: In order to capture the complex semantic correlation between images and texts, the present invention proposes a hierarchical modal fusion method based on a cross-modal attention mechanism. 1 ,v 2 ,…,v K}∈R K*D and text word W = {w 1 ,w 2 ,…,w L}∈R L*D The mutual attention mechanism between them captures all potential local semantic alignments. The text representation that receives attention from each region is obtained The calculation formula is as follows:

[0027] W * =Multihead(V,W,W)=Concat(head 1 ,...,head i ,...,head H )Z O

[0028]

[0029] Among them, W * ∈R K*D , Z O ∈R D*D , H is the number of attention heads. Concat(·) is the concatenation operation on the feature dimension D. Att(·) is the scaled dot product attention, calculated as follows:

[0030]

[0031] Similarly, we obtain the image representation that each word pays attention to The calculation formula is as follows:

[0032] V * =Multihead(W,V,V)=Concat(head 1 ,...,head i ,...,head H )Z O

[0033]

[0034] Among them, V * ∈R L*D , Z O ∈R D*D , H is the number of attention heads. Concat(·) is the concatenation operation on the feature dimension D. Att(·) is the scaled dot product attention.

[0035] In addition, the whole image and text are mapped into a common space to achieve global semantic alignment. Through the learnable embedding matrix X w Mapping, global representation of text W g The calculation formula is as follows:

[0036]

[0037] Through the learnable embedding matrix X v and X g Mapping, global representation of image V g The calculation formula is as follows:

[0038]

[0039] V g2 =X g T G

[0040]

[0041] in, σ is the sigmoid function, X t and b are the parameters of the fully connected layer, and Cat(·) is the concatenation operation.

[0042] The image-text similarity calculation stage corresponds to step (4) of the technical solution. The specific implementation method is as follows: the matching score between image I and text T consists of two parts: local similarity score and global similarity score. * and word attention image representation V * , the local similarity calculation formula is as follows:

[0043]

[0044] Among them, u 1 is a hyperparameter. R(﹒) is the dot product similarity, and the calculation formula is defined as follows:

[0045]

[0046] At the same time, the global similarity calculation formula between image I and text T is as follows:

[0047] S global (I,T)=R(V g ,W g )

[0048] Based on the local similarity score S local (I,T) and the global similarity score S global (I, T), the similarity score S(I, T) between image I and text T can finally be defined as:

[0049] S(I,T)=u 2 ·S local (I,T)+(1-u 2 )·S global (I,T)

[0050] Among them, u 2 is a hyperparameter.

[0051] The contrastive learning training phase corresponds to step (5) of the technical solution. The specific implementation method is as follows: the present invention uses a contrastive learning algorithm based on image-text similarity to train a model, constraining the similarity of semantically matched image-text pairs to be greater than the similarity of other semantically mismatched image-text pairs.

[0052] During model training, assume that each batch has N image-text pairs. Step (2) randomly adopts the second data augmentation strategy of adding Gaussian noise, shuffle, cutoff and random discarding of semantic features to the word sequence vector corresponding to each input text in the batch, and generates N new texts Thus, for each matching image-text pair (I,T) in the batch, the text T has N-1 unmatched images from the original dataset. The corresponding contrastive learning loss function formula for the text retrieval image task is as follows:

[0053]

[0054] Among them, τ is a hyperparameter. In the process of loss optimization, the desired result is e S(I,T) / τ The value is as large as possible, that is, the distance between the semantically matched image and text is as close as possible. The value should be as low as possible, that is, the distance between semantically mismatched images and texts should be as far as possible. From the formula, when minimizing L T2I In the process of (I, T), we need to make the formula close to 0, that is, to make the fraction inside log close to 1. To achieve this effect, we should make It can be found that it is consistent with the training objectives.

[0055] For each matching image-text pair (I,T) in the batch, image I has N-1 unmatched texts from the original dataset N newly generated unmatched texts The corresponding contrastive learning loss function formula for the image retrieval text task is as follows:

[0056]

[0057] Among them, if the similarity score of the image-text pair If it exceeds the pair S(I,T), then α n Set to zero, otherwise α n Set to 1.

[0058] Step (2) randomly adopts the first data augmentation strategy of concatenating text and cutting text for each input text in the batch to generate N new texts The word sequence vectors corresponding to these N texts are randomly augmented again using the second data augmentation strategy of adding Gaussian noise, shuffling, cutting off, and randomly discarding semantic features, and N new texts are generated. Therefore, for each newly generated matching image-text pair (I, T + ), text T +There are N-1 mismatched images from the original dataset The corresponding contrastive learning loss function formula for the text retrieval image task is as follows:

[0059]

[0060] For each newly generated matching image-text pair (I, T + ), image I has N-1 origins Unmatched text, N newly generated unmatched text The corresponding contrastive learning loss function formula for the image retrieval text task is as follows:

[0061]

[0062] Finally, if Figure 2 As shown in the figure, for each batch of N image-text pairs, the training objective function of the image-text similarity contrast learning algorithm is:

[0063]

[0064] This paper proposes a method for image-text retrieval based on contrastive learning and modal fusion. In order to test the effectiveness of this method, the R@1, R@5, and R@10 indicators are used to evaluate the two commonly used image-text retrieval datasets, MSCOCO and Flickr30K, and compared with other image-text retrieval methods. The R@1 / 5 / 10 recall rate refers to the ratio of the number of relevant results retrieved from the top-1 / 5 / 10 results to the number of all relevant results in the dataset, which measures the recall rate of the retrieval system.

[0065] The MSCOCO dataset contains 123,287 images and 616,767 texts, and each image has about 5 semantically corresponding sentences. It is divided into 113,287 training images, 5,000 validation images, and 5,000 test images. The Flickr30K dataset contains 31,783 images, and each image has 5 semantically corresponding sentences. It is divided into 29,783 training images, 1,000 validation images, and 1,000 test images.

[0066] The model uses Adam as the optimizer. During model training, the number of samples in each batch is set to 64, and the epoch is set to 20. The number of salient regions extracted by Faster-RCNN is set to 36, and the feature dimension D of the salient regions is set to 4. v The dimension D of the common mapping space is set to 1024. The hyperparameter u 1and λ are set to 0.5. For the MSCOCO dataset, the learning rate is set to 5e-4, decaying 10 times every 10 epochs, the hyperparameter τ is set to 0.05, and the hyperparameter u 2 Set to 0.8. For the Flickr30K dataset, the learning rate is set to 2e-4, decaying 10 times every 10 epochs, the hyperparameter τ is set to 0.01, and the hyperparameter u is 2 Set to 0.6.

[0067] Table 1 shows the comparison results of various retrieval indicators of different image and text retrieval models on the MSCOCO dataset; Table 2 shows the comparison results of various retrieval indicators of different image and text retrieval models on the Flickr30K dataset.

[0068] Table 1 Experimental results of different image-text retrieval models on the MSCOCO dataset

[0069]

[0070] Table 2 Experimental results of different image and text retrieval models on the Flickr30K dataset

[0071]

[0072]

[0073] From the experimental results in the above table, it can be seen that the image-text retrieval method based on contrastive learning and modal fusion proposed in this patent has achieved the best retrieval recall rate performance on the MSCOCO and Flickr30K datasets compared with other image-text retrieval methods, which shows that the present invention can effectively improve the accuracy of image-text retrieval tasks.

Claims

1. A method for image-text retrieval based on contrastive learning and modality fusion, comprising the following steps: (1) For the input image-text pair, use the object detection model to encode the image and obtain the salient area features of each image; Use a deep residual network to encode the image and obtain the global features of each image; Use the pre-trained language model to encode the word sequence corresponding to the text to obtain the word vector of each word; (2) For an input image-text pair, a first data augmentation strategy of changing the text at the original input layer is used to generate a positive sample pair, wherein the positive sample pair is an image-text pair with semantic matching, and a second data augmentation strategy of adding noise at the word sequence vector layer is used to generate a negative sample pair, wherein the negative sample pair is an image-text pair with semantic mismatch; (3) Perform cross-modal hierarchical semantic fusion of images and texts to eliminate the semantic gap between visual modalities and language modalities: Use a cross-modal attention mechanism to mine the alignment relationship between all image salient regions and text words to achieve cross-modal local semantic alignment of images and texts; Use a learnable embedding matrix to map the entire image and text to a common subspace to achieve cross-modal global semantic alignment of images and texts; (4) The similarity of an image-text pair is determined by both local semantic similarity and global semantic similarity. The local semantic similarity is learned by taking fine-grained image salient regions and text words as objects, and the global semantic similarity is learned by taking the entire image and text as objects. (5) Combining the positive sample pairs and negative sample pairs generated in step (2), using a contrastive learning loss function, constraining the similarity of semantically matching image-text pairs to be greater than the similarity of other semantically mismatched image-text pairs, and continuously optimizing the training model.

2. According to claim 1, a method for image-text retrieval based on contrastive learning and modality fusion, characterized in that: In the step (1), the image salient area features are obtained by mapping through the target detection model Faster-RCNN, the image global features are obtained by mapping through the deep residual network ResNet152, and the word sequence vector is obtained by mapping through the pre-trained language model BERT.

3. According to claim 1, the image-text retrieval method based on contrastive learning and modality fusion is characterized in that: In the step (2), the first data augmentation strategy is text splicing and text cropping; the second data augmentation strategy is adding Gaussian noise, disordering, truncation and random discarding of semantic features.

4. The image-text retrieval method based on contrastive learning and modality fusion according to claim 1, characterized in that: In the step (3), a scaled dot product attention mechanism is used to achieve local semantic alignment and cross-modal hierarchical semantic fusion of image salient areas and text words, and then obtain new global features and local features of the image and text.

5. The image-text retrieval method based on contrastive learning and modality fusion according to claim 1, characterized in that: In the step (4), a dot product similarity calculation method is used to obtain global semantic similarity and local semantic similarity.

6. The image-text retrieval method based on contrastive learning and modality fusion according to claim 1, characterized in that: In the step (5), the positive sample pairs and negative sample pairs constructed in step (2) are combined, and the contrastive learning loss function InfoNCE is used to calculate the loss, so as to shorten the distance between the positive sample pairs and extend the distance between the negative sample pairs in the semantic space.

Images