A text and image retrieval method and system based on asymmetric dual encoders

Abstract

This invention discloses an image-text retrieval method and system based on asymmetric dual encoders. The method uses dual encoders and a cross-modal encoder to encode images and / or text respectively. Similarity scores are obtained by calculating the cosine similarity of the encoded features of the image and / or text. The similarity scores are then ranked, and the image-text with the highest score is selected as the output, thus achieving image-text retrieval. On one hand, the asymmetric dual encoder retains the cross-attention mechanism, achieving the same high retrieval performance as the cross-modal encoder. On the other hand, the asymmetric dual encoder independently encodes the query and candidate features, enabling the model to overcome batch size limitations and mine more information examples. Batch size significantly affects encoder learning, effectively solving the problems of reduced accuracy caused by pure dual encoder models and low computational efficiency caused by pure cross-modal encoders.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and natural language processing technology, specifically relating to a text and image retrieval method and system based on asymmetric dual encoders. Background Technology

[0002] With the rapid development of the internet, more and more data is being uploaded and shared among numerous users. This data is presented in different forms, such as images, text, videos, and audio, and they appear simultaneously, expressing similar semantic meanings. This phenomenon has further driven the demand for matching between different modalities, especially image and text matching, which are the two most frequently occurring modalities on the internet. Therefore, image-text retrieval has attracted much attention in the fields of computer vision and natural language processing.

[0003] Most vision and language tasks rely on cross-modal understanding to bridge the semantic gap between visual and textual cues in images and text. To address this, two approaches are typically used: cross-modal encoders that utilize coupled image-text pairs with cross-attention mechanisms, and dual encoders that encode images and text separately without interaction. Dual encoders typically learn high-dimensional vectors with semantic information by encoding each image and each piece of text as input. In contrast, cross-modal encoders learn contextualized representations of image-text pairs by encoding each image-text pair with information from the encoder. Considering both speed and accuracy, dual encoders are faster but less accurate, while cross-modal encoders are slower but more accurate. Summary of the Invention

[0004] This invention addresses the problems of low accuracy and slow speed of dual encoders and cross-modal encoders in existing technologies by providing an image-text retrieval method and system based on asymmetric dual encoders. It uses both dual encoders and a cross-modal encoder to encode images and / or text. Similarity scores are obtained by calculating the cosine similarity of the encoded features of the image and / or text, and the similarity scores are ranked. The image-text retrieval with the highest score is selected as the output, thus achieving image-text retrieval. On the one hand, the asymmetric dual encoder retains the cross-attention mechanism, achieving the same high retrieval performance as the cross-modal encoder. On the other hand, the asymmetric dual encoder independently encodes query and candidate features, enabling the model to overcome batch size limitations and mine more information examples. Batch size significantly affects encoder learning, effectively solving the problems of reduced accuracy caused by pure dual encoder models and excessively low computational efficiency caused by pure cross-modal encoders.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is: a text-image retrieval method based on asymmetric dual encoders, which uses dual encoders and cross-modal encoders to encode images and / or text respectively, obtains similarity scores by calculating the cosine similarity of the features after encoding the images and / or text, sorts the similarity scores, and selects the image and text with the highest score as the output to achieve text-image retrieval.

[0006] As an improvement to the present invention, a text and image retrieval method based on asymmetric dual encoders includes the following steps:

[0007] S1: For a text and image dataset with S training samples, randomly sample N samples as a batch. The i-th text and image sample pair {I} i T i} in, I i Let T represent the i-th image. i This represents the i-th text;

[0008] S2: Perform data augmentation on the image and text, and then process the input text T. i Divide the words into n words separated by spaces, and further encode them into n word embeddings W. i ={w1, w2, ..., w n}; Input image I i The image is divided into m regions and further encoded to obtain the image code R. i ={r1, r2, ... r m};

[0009] S3: Use a dual encoder to process the word embeddings W obtained in step S2. i and image encoding R i Feature extraction is performed to obtain dual-coded text features and dual-coded image features. The dual-coded text features are as follows:

[0010]

[0011] The dual-coded image features are:

[0012]

[0013] S4: Use a cross-modal encoder to process the word embeddings W obtained in step S2. i and image encoding R i Feature extraction is performed to obtain cross-modal coded text features and cross-modal coded image features. The cross-modal coded text features are as follows:

[0014]

[0015] The cross-modal coded image features are:

[0016]

[0017] S5: Apply the dual-encoded text features obtained in step S3 and the cross-modal encoded text features obtained in step S4 The final dual-encoded text features are obtained by aggregation. and cross-modal encoded text features The dual-coded image features obtained in step S3 and the cross-modal coded image features obtained in step S4 The final dual-coded image features are obtained by aggregation. and cross-modal coding image features

[0018] S6: Calculate the loss of a batch of data using the pagination loss function, update the model parameters by minimizing the loss function, train the model until the loss converges and the optimal model is obtained; otherwise, return to step S1.

[0019] S7: Input the image into the optimal cross-modal encoder model to obtain cross-modal encoded image features. The text to be matched is input into the optimal dual encoder model to obtain dual-encoded text features. The cosine similarity of the features after image and text encoding is calculated to obtain a similarity score. The similarity scores are sorted, and the image and text with the highest score is selected as the output to realize image and text retrieval.

[0020] As an improvement to the present invention, in step S2, the text T i The words are mapped and encoded as word embeddings through a learnable fully connected layer; Image I i After encoding using the Faster R-CNN model, the image encoding R is obtained. i The Faster R-CNN model is a model pre-trained on the Visual Genome dataset; the image data augmentation includes rotation transformation, scaling transformation, translation transformation, and scale transformation; the text data augmentation includes word synonym replacement.

[0021] As another improvement of the present invention, the dual encoder in step S3 is an encoder pre-trained with a BERT model.

[0022] The cross-modal encoder in step S4 is an encoder pre-trained with the BERT model, where the cat operation is to concatenate two vectors.

[0023] When an image is used to retrieve text, the BERT model uses the following computational method in its encoding process:

[0024]

[0025]

[0026]

[0027]

[0028] Among them, W Q W K W V These represent the weights of the mapping layer; Q, K, and V represent the query encoding, key encoding, and value encoding, respectively; d k Indicates the length of the query encoding.

[0029] As another improvement of the present invention, the aggregation operation in step S5 is to perform average aggregation on the feature vectors, that is:

[0030]

[0031]

[0032]

[0033]

[0034] in, Representing dual-encoded text features The encoding of the j-th word in the text; Representing cross-modal text features The encoding of the j-th word in the text; Representing dual-coded image features The encoding of the j-th part in; Representing cross-modal image features The encoding of the j-th part in .

[0035] As another improvement of the present invention, step S6 further includes sampling additional negative samples for training, wherein the loss function is:

[0036]

[0037] Where M is a negative sample dataset randomly drawn outside the batch data; S(,) is a function for calculating feature similarity.

[0038] To achieve the above objectives, the present invention also adopts the following technical solution: a text and image retrieval system based on asymmetric dual encoders, comprising a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0039] Compared with existing technologies, this invention provides a text and image retrieval method based on asymmetric dual encoders. This method allows for the introduction of more negative samples during batch training. The asymmetric dual encoder of this invention combines a cross-modal encoder and a dual encoder, enabling the model to learn better semantic representations during training. During each retrieval, the query is encoded using the dual encoder, and the candidate is encoded using the cross-modal encoder, achieving higher accuracy while maintaining similar speed to the cross-modal encoder. The method proposed in this invention demonstrates good performance on public datasets. Attached Figure Description

[0040] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0041] Figure 2 This is a network structure diagram of the method of the present invention;

[0042] Figure 3 This is a schematic diagram comparing the test example of the present invention with the baseline method;

[0043] Figure 4 This is a schematic diagram comparing the test examples of the present invention with other methods. Detailed Implementation

[0044] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0045] Example 1

[0046] This embodiment provides an image-to-text retrieval method based on asymmetric dual encoders, specifically for image-to-text retrieval on the MSCOCO and Flickr30K datasets. For image-to-text retrieval, a dual encoder is used to encode the text, and a cross-modal encoder is used to encode the image. Conversely, for text-to-image retrieval, a dual encoder is used to encode the image, and a cross-modal encoder is used to encode the text for text-to-image retrieval. Finally, their cosine similarity is calculated to obtain their similarity scores. The similarity scores are then sorted, and the image or text with the highest score is selected as the final output. The entire network structure diagram is shown below. Figure 2 As shown.

[0047] A text and image retrieval method based on asymmetric dual encoders, such as Figure 1 As shown, it includes the following steps:

[0048] S1: For a text and image dataset with S training samples, randomly sample N samples as a batch. The i-th text and image sample pair {I} iT i} in, I i Let T represent the i-th image. i This represents the i-th text;

[0049] S2: Perform data augmentation on the image and text, and then process the input text T. i Divide the words into n words separated by spaces, and further encode them into n word embeddings W. i ={w1, w2, ..., w n}; Input image I i The image is divided into m regions and further encoded to obtain the image code R. i ={r1, r2, ... r m};Text T i The words are mapped and encoded as word embeddings through a learnable fully connected layer; Image I i After encoding using the Faster R-CNN model, the image encoding R is obtained. i The Faster R-CNN model is a model pre-trained on the Visual Genome dataset; the image data augmentation includes rotation transformation, scaling transformation, translation transformation, and scale transformation; the text data augmentation includes word synonym replacement.

[0050] S3: Use a dual encoder to process the word embeddings W obtained in step S2. i and image encoding R i Feature extraction was performed to obtain dual-encoded text features and dual-encoded image features. The dual encoder is an encoder pre-trained with a BERT model. The dual-encoded text features are as follows:

[0051]

[0052] The dual-coded image features are:

[0053]

[0054] The calculation method used in the BERT model encoding process (taking image-to-text retrieval as an example) is as follows:

[0055]

[0056]

[0057]

[0058]

[0059] S4: Use a cross-modal encoder to process the word embeddings W obtained in step S2. iand image encoding R i Feature extraction is performed to obtain cross-modal encoded text features and cross-modal encoded image features. The cross-modal encoder is an encoder pre-trained with a BERT model, where the `cat` operation concatenates two vectors. The cross-modal encoded text features are as follows:

[0060]

[0061] The cross-modal coded image features are:

[0062]

[0063] The calculation method for CE is the same as that for BE.

[0064] S5: Apply the dual-encoded text features obtained in step S3 and the cross-modal encoded text features obtained in step S4 The final dual-encoded text features are obtained by aggregation. and cross-modal encoded text features The dual-coded image features obtained in step S3 and the cross-modal coded image features obtained in step S4 The final dual-coded image features are obtained by aggregation. and cross-modal coding image features The aggregation operation is to average and aggregate the feature vectors, that is:

[0065]

[0066]

[0067]

[0068]

[0069] S6: Calculate the loss for a batch of data using a folding loss function, ensuring that text features and image features within the same text pair are as close as possible, while text features and image features across different text pairs are as far apart as possible. In addition, sample extra negative samples from the entire dataset for training, ensuring that all data in the batch are far removed from the extra-sampled negative samples; the loss function can be expressed as:

[0070]

[0071]

[0072] Where M is a dataset of negative samples randomly selected outside the batch data; S(,) is a function for calculating feature similarity. This invention employs an asymmetric dual-encoder image-text retrieval method. For the input object, a cross-modal encoder is used for feature extraction to maintain its semantic interaction, while for the object to be matched, a dual encoder is used for feature extraction to reduce memory usage. Therefore, compared to a pure cross-modal encoder, this invention has more negative samples for training in a batch, and compared to a pure dual encoder, it has richer interactive semantic features.

[0073] Minimize the loss function and update the model parameters. If the loss converges, stop training; otherwise, return to step S1. The method used to minimize the loss is the Adam algorithm.

[0074] S7: Input the image into the optimal cross-modal encoder model to obtain cross-modal encoded image features. The text to be matched is input into the optimal dual encoder model to obtain dual-encoded text features. Calculate the similarity, then calculate the similarity between cross-modal encoded image features and dual-encoded text features for each text within the retrieval range, sort them, and select the text with the highest similarity as the output result. Image retrieval follows the same principle.

[0075] Test case

[0076] In this test case, the MSCOCO dataset consists of 123,287 images containing common natural and everyday objects, with each image containing 5 descriptive texts. The Flickr30K dataset contains 31,783 images, with each image containing 5 descriptive texts.

[0077] During model training, this invention uses the ADAM optimizer to train the model for 10 epochs with a learning rate of 0.001, then reduces it to 0.0001 for another 15 epochs. For the encoder input, the input sequence length for both the text and image encoders is set to 32. The dual encoders share the weights of both the text and image encoders. Furthermore, the margin α in the loss function is set to 0.2.

[0078] This invention uses the Faster R-CNN model to extract region-of-interest features for each image, and all Transformer encoders load pre-trained models from BERT models. This invention saves memory and computational costs during training by applying a miniature BERT model with only 3 Transformer layers, as experiments have shown that the 3-layer model achieves similar accuracy to the 12-layer model.

[0079] We compared some recent methods based on pure dual encoders and cross-modal encoders on the MSCOCO and Flickr30K datasets. The experimental results are as follows: Figure 3 As shown, T-to-I R@K represents the accuracy of retrieving images from text with the correct results in the first K outputs, and I-to-T R@K represents the accuracy of retrieving text from images with the correct results in the first K outputs. Figure 3 As can be seen from the above, the image and text retrieval method based on asymmetric dual encoder proposed in this invention has higher accuracy than methods based on dual encoder and cross-modal encoder.

[0080] In addition, it was compared with some other existing methods, such as Figure 4 As shown. Figure 4 As can be seen, this invention achieves better accuracy compared to robust cross-modal baselines. Note that ViLT and ViLBERT are pre-trained on large-scale multimodal datasets, while this invention is not pre-trained on these datasets. Pre-training could further improve the model in this invention. Furthermore, this invention does not use labels from OSCAR. Further comparisons with methods including dual-encoder-based and attention-based methods show that the accuracy of this invention is significantly higher than these methods. Therefore, the algorithm proposed in this invention has significant advantages in practical applications.

[0081] In summary, the asymmetric dual encoder of this invention combines a cross-modal encoder and a dual encoder. For each retrieval, the query is encoded using the dual encoder, and the candidate is encoded using the cross-modal encoder. For image-to-text retrieval, the text is encoded using the dual encoder, and the image is encoded using the cross-modal encoder. Conversely, for text-to-image retrieval, the image is encoded using the dual encoder, and the text is encoded using the cross-modal encoder for text-to-image retrieval. On the one hand, the asymmetric dual encoder retains the cross-attention mechanism, achieving the same high retrieval performance as the cross-modal encoder; on the other hand, the asymmetric dual encoder independently encodes the query and candidate features, enabling the model to overcome batch size limitations and mine more information examples, effectively solving the problems of reduced accuracy caused by pure dual encoder models and excessively low computational efficiency caused by pure cross-modal encoders.

[0082] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for retrieving images and texts based on asymmetric double encoder, characterized in that: The image and / or text are encoded using a dual encoder and a cross-modal encoder respectively. A similarity score is obtained by calculating the cosine similarity of the encoded features of the image and / or text. The similarity scores are then sorted, and the image and text with the highest score is selected as the output to achieve image-text retrieval. The specific steps include: S1: For a training sample size of... The image and text dataset, randomly sampled Each sample is considered as a batch, and the first... A pair of image and text samples middle, Indicates the first One image, Indicates the first One text; S2: Perform data augmentation on images and text, and then input the text. Divided into spaces 1 word, and further encoded as Word embedding ; Input image Divided into Each region is further encoded to obtain the image code. ; S3: Use a dual encoder to process the word embeddings obtained in step S2. and image encoding Feature extraction is performed to obtain dual-coded text features and dual-coded image features. The dual-coded text features are as follows: The dual-coded image features are: ; S4: Use a cross-modal encoder to process the word embeddings obtained in step S2. and image encoding Feature extraction is performed to obtain cross-modal coded text features and cross-modal coded image features. The cross-modal coded text features are as follows: ; The cross-modal coded image features are: ; S5: Apply the dual-encoded text features obtained in step S3 and the cross-modal encoded text features obtained in step S4 The final dual-encoded text features are obtained by aggregation. and cross-modal encoded text features ; The dual-coded image features obtained in step S3 and the cross-modal coded image features obtained in step S4 The final dual-coded image features are obtained by aggregation. and cross-modal coding image features ; S6: Calculate the loss of a batch of data using the pagination loss function, update the model parameters by minimizing the loss function, train the model until the loss converges and the optimal model is obtained; otherwise, return to step S1. S7: Input the image into the optimal cross-modal encoder model to obtain cross-modal encoded image features. The text to be matched is input into the optimal dual encoder model to obtain dual-encoded text features. The cosine similarity of the features after image and text encoding is calculated to obtain a similarity score. The similarity scores are sorted, and the image and text with the highest score is selected as the output to realize image and text retrieval.

2. The image and text retrieval method based on asymmetric dual encoder as described in claim 1, characterized in that: In step S2, the text The words are mapped and encoded as word embeddings through a learnable fully connected layer; images By using After model encoding, image encoding is obtained. The Faster R-CNN model is a model pre-trained on the VisualGenome dataset; image data augmentation includes at least rotation transformation, scaling transformation, translation transformation, and scale transformation; text data augmentation includes word synonym replacement.

3. The image and text retrieval method based on asymmetric dual encoder as described in claim 1, characterized in that: The dual encoders in step S3 are encoders pre-trained with the BERT model. The cross-modal encoder in step S4 is an encoder pre-trained with a BERT model, wherein... The operation involves concatenating two vectors. When retrieving text from an image, the calculation method used in the encoding process is as follows: ； ； ； ； in, , , These represent the weights of the mapping layer; , , These represent query encoding, key encoding, and value encoding, respectively. Indicates the length of the query encoding.

4. The image and text retrieval method based on asymmetric dual encoder as described in claim 2 or 3, characterized in that, The aggregation operation in step S5 is to perform average aggregation on the feature vectors, that is: ； ； ； ； in, Representing dual-encoded text features The first in The encoding of each word; Representing cross-modal text features The first in The encoding of each word; Representing dual-coded image features The first in The encoding of each part; Representing cross-modal image features The first in The encoding of each part.

5. The image and text retrieval method based on asymmetric dual encoder as described in claim 4, characterized in that: Step S6 further includes sampling additional negative samples for training, and the loss function is: ； in, This is a dataset of negative samples randomly drawn outside of the batch data; The function is used to calculate feature similarity.

6. A text and image retrieval system based on asymmetric dual encoders, comprising a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-5 above.

Images