Combined image retrieval method based on picture mapper and text self-occlusion mapper

Abstract

The combined image retrieval method based on the combination of the picture mapper and the text self-mask mapper comprises the following steps: first, performing scaling, noise adding and other operations on the input picture, and then inputting the picture into a frozen pre-training image encoder to obtain image features; converting the image features into pseudo word marks s1 by using an image mapper; converting the image features into pseudo word marks s2 by using a text self-mask mapper; respectively replacing mark symbols in a pre-provided prompt text with the pseudo word marks s1 and s2; and splicing the input modified text to the prompt text to obtain two new texts x1 and x2. The texts x1 and x2 are input into a frozen pre-training text encoder to obtain two features, the two features are fused by averaging, and finally, the obtained average feature is compared with a target image. The combined image retrieval method solves the shortcomings of using expensive triplets to train a combined image retrieval model, and improves the generalization ability of the model for the downstream task of combined image retrieval.

Description

Technical Field

[0001] This invention relates to the field of multimodal retrieval based on text and images, and particularly to a method using an image mapper and a text self-masking mapper. Background Technology

[0002] Combined text and image retrieval aims to retrieve matching target images from a given original image and edited text describing that image. This approach offers greater flexibility and a better user experience for image-based search engine applications (such as e-commerce platforms and internet search), as language is the most natural method of human-computer interaction. Compared to traditional methods that only use text to search for target images or use images to search for similar images, combined image retrieval integrates information from both text and images, enabling more accurate retrieval of target images in most cases.

[0003] In combined image retrieval tasks, most traditional models rely on post-processing image-text feature fusion, which involves combining visual and linguistic features extracted using different encoders. Existing visual-language foundational models (such as CLIP and ALIGN) are pre-trained on massive datasets of hundreds of millions of image-text pairs for both image and text encoders. Having been trained on vast amounts of data, these models have seen different texts used to describe an image, allowing them to handle diverse tasks. The pre-trained CLIP model performs exceptionally well, some methods utilize pre-trained BERT models to fuse image and text features, and others train a shallow transformer from scratch to fuse image and linguistic features directly at input. However, these methods ultimately require tuning their models to specific datasets for combined image retrieval tasks to fully leverage the combinatorial capabilities of pre-trained language models.

[0004] Datasets for combined image retrieval tasks primarily consist of triplets of the original image, query text, and target image. Unlike traditional image-text matching datasets, these triplet datasets are virtually impossible to collect via web crawlers, requiring expensive manual creation of each triplet. Therefore, existing triplet datasets are relatively small in scale and target specific, narrow domains. To improve the scalability and practicality of combined image retrieval tasks, optimizing the training data and training patterns of traditional models becomes crucial. Summary of the Invention

[0005] To overcome the limitations of combined image retrieval tasks, such as limited applicability and poor scalability when trained on existing datasets, this invention proposes a combined image retrieval method based on image mappers and text self-masking mappers to address the generalization problem of combined image retrieval. This method does not require expensive triple training datasets, only image titles and unlabeled image datasets, while achieving performance similar to methods trained using labeled triple data.

[0006] To achieve the above objectives, the present invention provides a combined image retrieval method based on an image mapper and a text self-masking mapper, employing the following technical solution:

[0007] Step 1: Read the dataset, using images and text descriptions as data inputs to the model;

[0008] Step 1.1: First, scale the image to the predetermined size and perform data augmentation such as horizontal flipping and random addition of Gaussian noise.

[0009] Step 1.2: Use the tag extractor in the natural language processing tool to extract keywords from the current text, add the extracted keywords to a list and remove duplicates to obtain a keyword list. This list records all the keywords contained in the current text.

[0010] Step 1.3: Traverse the keyword list and replace the keywords in the text with the '[$]' symbol to obtain a new text description.

[0011] Step 2, train the image mapper f M Transform image features into pseudo-language tag embeddings: Where s is the pseudo-language tag embedding. These are image features.

[0012] Step 2.1, pass the input image through the frozen visual encoder f I Image features are obtained through processing

[0013] Step 2.2, construct an image mapper f with parameter M. M .

[0014] Step 2.3, using image mapper f M Image features Convert to pseudoword tag s.

[0015] Step 2.4: Construct the text prompt: "A photo of", followed by an image feature. The converted pseudo-word tags s yield the concatenated text. “A photo of [s]”.

[0016] Step 2.5, combine the text Input frozen text encoder f φ Obtain the concatenated text features

[0017] Step 2.6, the training objective is to extract the features of the concatenated text. Able to fully represent the input image information, i.e., training the image mapper f M The ability to map image features to pseudoword tags. To achieve this, minimizing the contrastive loss relative to the mapper makes similar images and text closer together in the same embedding space, and dissimilar images and text further apart;

[0018] The functional expression for the contrast loss is as follows:

[0019]

[0020] in, This represents the contrast loss from text to image. This represents the contrast loss from image to text. τ is a temperature parameter used to scale the similarity score. p represents the number of samples in a training batch. i Yes Normalization is performed to obtain the result.

[0021] Step 3: Since the input of the text self-masking mapper φ is plain text data, and this invention aims to use it for combined image retrieval tasks, a method of adding noise is adopted to bridge the modal gap between plain text and images.

[0022] Step 3.1, input the text x c The frozen pre-trained text encoder f φ Obtain text features z c .

[0023] Step 3.2, analyze the text features z c Add noise n.

[0024] Step 4: Train the text self-masking mapper φ to capture information in the image through plain text training.

[0025] Step 4.1: Construct a text self-masking mapper φ with parameter N.

[0026] Step 4.2: Use the text self-masking mapper φ to extract the text features z that have had noise added in Step 3.2. c Convert to pseudoword tag

[0027] Step 4.3: Using the text description obtained in Step 1.3, replace the keywords with '[$]' symbols to mark the pseudo-words. Replace all the '[$]' symbols, and denote it as

[0028] Step 4.4, will Input to the frozen pre-trained text encoder f φ Obtain text features

[0029] Step 4.5, for the two obtained text features z c and The text self-masking map is trained by minimizing the mean squared error loss between the two.

[0030] The mean squared error loss function is expressed as follows:

[0031]

[0032] Step 5: Based on the two loss functions from Steps 2 and 4, optimize the model using the backpropagation algorithm and the gradient descent algorithm; specifically including:

[0033] Step 5.1: Based on the actual input and the expected output, obtain the overall error formula, which is as follows:

[0034]

[0035] In the formula It calculates the features of the concatenated text. Image features The contrast loss function, It is to calculate text features z c Text features after being replaced by pseudo-word tags The mean squared error loss function of the difference between them.

[0036] Step 5.2: Optimize model parameters using backpropagation and gradient descent algorithms; set the batch size to 64, use the Adam optimizer, and set the initial learning rate to 1*10^6. -5 The weights were decayed to 0.01, and Dropout with a probability of 50% was used for regularization. The training lasted for a total of 60 rounds.

[0037] Step 6, test the model.

[0038] Step 6.1 provides a prompt text, such as "a photo of [$]". The features obtained from the images in the dataset, processed by a frozen pre-trained visual encoder, are used as the query set.

[0039] Step 6.2: First, the images in the input data are processed by a frozen pre-trained visual encoder to extract features. Will The inputs are respectively fed into the image mapper f M In the text self-masking mapper φ, pseudo-words s1 and s2 are obtained respectively. These pseudo-words replace the '[$]' symbol in the pre-provided prompt text. The text part in the data is then concatenated to the prompt text to obtain two new texts x1 and x2.

[0040] Step 6.3: Input the texts x1 and x2 into the frozen pre-trained text encoder f, respectively. φ In the process, two features are obtained, and the two features are then fused.

[0041] Step 6.4: Calculate the cosine similarity between the fused features obtained in step 6.3 and the features in the query set, and then sort them to obtain the corresponding query results.

[0042] Preferably, in step 1.1, the image is first scaled to 224px*224px, and data augmentation is performed by horizontal flipping and randomly adding Gaussian noise.

[0043] Preferably, the natural language tool mentioned in step 1.2 is the spaCy natural language processing library, which extracts keywords as nouns ('NOUN' or 'PROPN') and adjectives ('ADJ').

[0044] Preferably, the replacement rule in step 1.3 is to replace all keywords in the original text with the symbol '[$]'.

[0045] Preferably, the visual encoder f described in step 2.1 I For ResNet networks, Vision Transformer networks.

[0046] Preferably, the image mapper f described in step 2.2 M It consists of a three-layer MLP containing 512 hidden units and uses the ReLU activation function, with approximately 800,000 parameters.

[0047] Preferably, the text encoder f described in step 2.5 φ It is a Transformer network.

[0048] Preferably, the noise added in step 3.2 is That is, a random scalar multiplied by a random vector drawn from a Gaussian distribution.

[0049] Preferably, the text self-masking mapper φ described in step 4.1 consists of a three-layer MLP and uses the GeLU activation function.

[0050] Preferably, step 5.2 specifically includes: setting the batch size to 64, using the Adam optimizer, and setting the initial learning rate of the network to 1*10^6. -5 The weights were decayed to 0.01, and Dropout with a probability of 50% was used for regularization. The training lasted for a total of 60 rounds.

[0051] Preferably, the fusion method described in step 6.3 is mean fusion.

[0052] The working principle of this invention is as follows: This invention primarily addresses the problems of expensive triplet datasets and poor generalization to downstream tasks in combined image retrieval tasks by employing two mapping networks. First, an image mapper is trained using images and simple cue text, enabling it to reconstruct visual representations within the text embedding space. Then, a text self-masking mapper is trained using text labels. This compensates for the lack of text diversity during image mapper training and requires only text for training. Training these two mappers requires only a large number of image-text pairs as input, eliminating the need for expensive triplet data. Furthermore, the large number of image-text pairs allows the model to learn knowledge from various domains, rather than being limited to a specific task, thus improving the model's generalization ability to downstream tasks.

[0053] The image mapper and text self-masking mapper proposed in this invention can both be extracted and trained as independent models, achieving good experimental results. Compared to the image mapper, the text self-masking mapper only uses text labels during training, making its training speed very fast. Compared to various complex multimodal retrieval models (with a time complexity of O(mn)), this invention employs a two-stream feature extraction network, requiring only text feature extraction and image feature extraction networks to extract text and image features respectively, resulting in a time complexity of O(m+n). Therefore, compared to traditional models for combined image retrieval tasks, this invention offers faster matching and retrieval performance.

[0054] The advantages of this invention are: 1. It provides a new training approach for the field of combined image retrieval. 2. The training data no longer relies on the expensive triplet datasets used in combined image retrieval tasks; training can be performed using only text captions and unlabeled image datasets. 3. It has better generalization ability, no longer targeting a specific domain, but can solve various combined image retrieval models. 4. In the zero-shot combined image retrieval task, on the FashionIQ dataset, the accuracy rank@10 is improved by 4.09%, and rank@50 by 4.28%. This is comparable to models trained using traditional triplet datasets. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the training process of the image mapper and the text self-masking mapper in the method of this invention.

[0056] Figure 2 This is the overall process for combined image retrieval in this invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and implementation methods.

[0058] A combined image retrieval method based on image mappers and text self-masking mapping, the training process is as follows: Figure 1 As shown, please follow these steps:

[0059] 1) Read the dataset and use each pair of matching images and image titles as data input to the model.

[0060] 11) First, scale the image to 224px*224px and perform data augmentation by horizontal flipping and randomly adding Gaussian noise.

[0061] 12) Use the tag extractor in the natural language processing tool spaCy to extract keywords from the current text. The keyword types are nouns ('NOUN' or 'PROPN') and adjectives ('ADJ'). Add the extracted keywords to a list and remove duplicates to obtain a keyword list that records all keywords contained in the current text.

[0062] 13) Iterate through the keyword list and replace all keywords in the text with the '[$]' symbol to obtain a new text description.

[0063] 2) such as Figure 1 As shown, the image mapper f is trained. M Transform image features into pseudo-language tag embeddings: Where s is the pseudo-language tag embedding. These are image features.

[0064] 21) The input image is processed by a frozen visual encoder f I Image features are obtained through processing Visual encoder f I The visual encoder Vision Transformer-base from the pre-trained CLIP is used, which mainly consists of normalized Norm layers, multi-head self-attention mechanism layers, and fully connected MLP layers.

[0065] 22) Construct an image mapper f M It consists of three MLP layers containing 512 hidden units and uses the ReLU activation function, with approximately 800,000 parameters.

[0066] 23) Using image mapper f M Image features Convert to pseudoword tag s:

[0067] 24) Construct a text prompt: “A photo of”, followed by an image feature. The converted pseudo-word tags s yield the concatenated text. “A photo of [s]”.

[0068] 25) Combine the text Input frozen text encoder f φ Obtain the concatenated text features Text encoder f φ The text encoder Transformer from the pre-trained CLIP is used. When the text encoder is ResNet, it mainly consists of convolutional CNN layers and residual blocks.

[0069] 26) The training objective is to obtain the features of the concatenated text. Able to fully represent the input image information, i.e., training the image mapper f M The ability to map image features to pseudoword tags. To achieve this, minimizing the contrastive loss relative to the mapper makes similar images and text closer together in the same embedding space, and dissimilar images and text further apart;

[0070] The functional expression for the contrast loss is as follows:

[0071]

[0072] in, This represents the contrast loss from text to image. This represents the contrast loss from image to text. τ is a temperature parameter used to scale the similarity score. p represents the number of samples in a training batch. i Yes Normalization is performed to obtain the result.

[0073] 3) Since the input of the text self-masking mapper φ is plain text data, and this invention is to use it for combined image retrieval tasks, a method of adding noise is adopted to bridge the modal gap between plain text and images.

[0074] 31) Input text x c The frozen pre-trained text encoder f φ Obtain text features z c .

[0075] 32) Regarding text features z c Add noise This is a random scalar value multiplied by a random vector drawn from a Gaussian distribution. Here, Unif(0,1) represents a uniform distribution, meaning that the scalar value is between 0 and 1, and any value has an equal probability of occurring. It follows a normal distribution, and each component is drawn independently from a normal distribution with a mean of 0 and a variance of 1.

[0076] 4) Train the text self-masking mapper φ to capture information in the image by training φ with plain text.

[0077] 41) Construct a text self-masking mapper φ with parameter N, consisting of three MLP layers, using the GeLU activation function.

[0078] 42) Use the text self-masking mapper φ to extract the text features z that have been noise-added in step 3.2. c Convert to pseudoword tag

[0079] 43) Using the text description obtained in step 1.3, replace the keywords with the '[$]' symbol to mark the pseudo-words. Replace all the '[$]' symbols, and denote it as

[0080] 44) will Input to the frozen pre-trained text encoder f φ Obtain text features

[0081] 45) For the two obtained text features z c and The text self-masking mapper φ is trained by minimizing the mean squared error loss between the two.

[0082] The mean squared error loss function is expressed as follows:

[0083]

[0084] 5) Based on the two loss functions in steps 2 and 4, optimize the model using the backpropagation algorithm and the gradient descent algorithm.

[0085] 51) Based on the actual input and the expected output, the overall error formula is obtained, which is:

[0086]

[0087] In the formula It calculates the features of the concatenated text. Image features The contrast loss function, It is to calculate text features z c Text features after being replaced by pseudo-word tags The mean squared error loss function of the difference between them.

[0088] 52) Optimize model parameters using backpropagation and gradient descent algorithms; set the batch size to 64, use the Adam optimizer, and set the initial learning rate of the network to 1*10. -5 The weights were decayed to 0.01, and Dropout with a probability of 50% was used for regularization. The training lasted for a total of 60 rounds.

[0089] 6) Test the model on the FashionIQ dataset, such as... Figure 2 As shown, the queried data consists of two parts: an image and modification instructions. This invention designs a prompt text template for this dataset as: "a photo of [$] + modification instructions".

[0090] 61) The images in the input data are first processed by a frozen pre-trained visual encoder to extract features. Will The inputs are respectively fed into the image mapper f M In the text self-masking mapper φ, pseudo-words s1 and s2 are obtained respectively. These pseudo-words replace the '[$]' symbol in the pre-provided prompt text. The text part in the data is then concatenated to the prompt text to obtain two new texts x1 and x2.

[0091] 62) Input the texts x1 and x2 into the frozen pre-trained text encoder f respectively. φIn the process, two features are obtained, and their means are fused. The resulting fused features are then compared with the features in the query set to calculate cosine similarity. The similarity scores are then ranked to obtain the top fifty images with the highest matching rates for each image-text input pair. These fifty images are then compared with the actual answers to calculate accuracy-related metrics.

[0092] 63) To verify the accuracy and effectiveness of the method of the present invention in practical application, the present invention calculates the accuracy rank@10 and rank@50 values ​​on the FashionIQ dataset in detail to evaluate the performance of the algorithm and presents the experimental results in Table 1. The following conclusions can be drawn from the experimental results: (1) The accuracy is significantly improved. After using the mapping model of the present invention, there is a significant improvement on the FashionIQ dataset. Compared with the zero-shot combination image retrieval task, the accuracy rank@10 is improved by 4.09% and rank@50 is improved by 4.28%. Compared with the model trained on the FashionIQ dataset, there is a significant improvement in the Shirt and Toptee classes, and the overall effect is on par with it. (2) After the model processes the image to obtain pseudo-words through the image mapper and the self-masking text mapper, it uses the text encoder to extract features for retrieval. This shows that the model of the present invention can indeed enable the mapper to extract image information. Furthermore, since it is trained using a large number of image-text pairs, its performance is comparable to that of models trained specifically on the FashionIQ dataset, which also demonstrates that the invented model has good generalization ability for downstream tasks in various fields.

[0093] Table 1: Search results on the FashionIQ dataset

[0094]

[0095] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An image retrieval method based on a combination of image mapper and text self-masking mapper, characterized in that, Training an image mapper and a text self-masking mapper using images and image caption text includes the following steps: Step 1: Read the dataset, using the images and image captions as input to the model; specifically including: Step 1.1: First, scale the image to the predetermined size, and then perform horizontal flipping and random Gaussian noise addition for data augmentation; Step 1.2: Use the tag extractor in the natural language tool to extract keywords from the image title text, add the extracted keywords to a list and remove duplicates to obtain a keyword list. This list records all the keywords contained in the image title text. Step 1.3: Iterate through the keyword list and replace the keywords in the image title text with '[ The '] symbol is used to obtain a new text description; Step 2, train the image mapper Transform image features into pseudo-language tag embeddings: Where s is the pseudo-language tag embedding. Image features; specifically including: Step 2.1: Pass the input image through the frozen visual encoder. Image features are obtained through processing ; Step 2.2, construct an image mapper with parameter M. ; Step 2.3, using an image mapper Image features Convert to pseudoword tag s; Step 2.4, construct the text prompt: "A photo of", followed by the image features. The converted pseudo-word tags s yield the concatenated text. "A photo of [s]"; Step 2.5, combine the text Input frozen text encoder Obtain the concatenated text features : ; Step 2.6 minimizes the contrast loss relative to the mapper so that similar images and texts are closer together in the same embedding space, and dissimilar images and texts are further apart; The functional expression for the contrast loss is as follows: in, This represents the contrast loss from text to image. This represents the contrast loss from image to text. It is a temperature parameter used to scale the similarity score. This indicates the number of samples in a training batch. Yes Normalization yields the result; Step 3 involves adding noise to bridge the modal gap between plain text and images; specifically, this includes: Step 3.1, input the text Frozen pre-trained text encoder Obtain text features ; Step 3.2, analyze text features Add noise n; Step 4, train the text self-masking mapper It captures information from images through training with plain text; specifically including: Step 4.1: Construct a text self-masking mapper with parameter N. ; Step 4.2, using a text self-masking mapper The text features with added noise in step 3.2 Convert to pseudoword tag : ; Step 4.3, using the text obtained in Step 1.3 corresponding to the image title text, using '[ The '] symbol replaces the text description of the keyword, marking pseudo-words. Replace all of the '[ The ]' symbol is denoted as ; Step 4.4, will Input to the frozen pre-trained text encoder Obtain text features ; Step 4.5, for the two obtained text features and To train the text self-masking mapping, the mean squared error loss between the two is minimized. The mean squared error loss function is expressed as follows: Step 5: Based on the two loss functions in Steps 2 and 4, optimize the model using the backpropagation algorithm and the gradient descent algorithm; Step 6, test the model.

2. The image retrieval method based on a combination of image mapper and text self-masking mapper as described in claim 1, characterized in that: Step 5 specifically includes: Step 5.1: Based on the actual input and the expected output, obtain the overall error formula, which is as follows: In the formula It calculates the features of the concatenated text. Image features The contrast loss function, It is to calculate text features Text features after being replaced by pseudo-word tags The mean squared error loss function for the differences between them; Step 5.2: Optimize model parameters using backpropagation and gradient descent algorithms; set the batch size to 64, use the Adam optimizer, and set the initial learning rate of the network to 1. 10 -5 The weights were decayed to 0.01, and Dropout with a probability of 50% was used for regularization. The training lasted for a total of 60 rounds.

3. The image retrieval method based on a combination of image mapper and text self-masking mapper as described in claim 1, characterized in that: Step 6 specifically includes: Step 6.1: Provide prompt text, using the features obtained from the images in the dataset after passing through the frozen pre-trained visual encoder as the query set; Step 6.2: First, the images in the input data are processed by a frozen pre-trained visual encoder to extract features. ,Will Input into the image mapper respectively and text self-masking mapper In the process, pseudo-words s1 and s2 are obtained respectively, and they are used to replace the '[' in the pre-provided prompt text. The '] symbol is used to append the text portion of the data to the prompt text, resulting in two new texts. , ; Step 6.3, transfer the text , The inputs are fed into the frozen pre-trained text encoder. In the process, two features are obtained, and the two features are then fused. Step 6.4: Calculate the cosine similarity between the fused features obtained in step 6.3 and the features in the query set, and then sort them to obtain the corresponding query results.

4. The image retrieval method based on a combination of image mapper and text self-masking mapper as described in claim 1, characterized in that: In step 1.1, first resize the image to 224px. The size is 224px, and it is horizontally flipped and data augmentation is performed by randomly adding Gaussian noise; The natural language tool mentioned in step 1.2 is the spaCy natural language processing library, which extracts keywords as nouns or adjectives; Step 1.3 The replacement rule is to replace all keywords in the image title text with '[ ]'symbol.

5. The image retrieval method based on a combination of image mapper and text self-masking mapper as described in claim 1, characterized in that: The visual encoder described in step 2.1 For ResNet networks, Vision Transformer networks; The image mapper described in step 2.2 It consists of a three-layer MLP containing 512 hidden units and uses the ReLU activation function; The text encoder described in step 2.5 It is a Transformer network.

6. The image retrieval method based on a combination of image mapper and text self-masking mapper as described in claim 2, characterized in that: The noise added in step 3.2 is That is, a random scalar multiplied by a random vector drawn from a Gaussian distribution; where Indicates a uniform distribution, meaning that the scalar value is between 0 and 1, and any value has the same probability of occurring. It follows a normal distribution, and each component is drawn independently from a normal distribution with a mean of 0 and a variance of 1; Step 4.1 Text self-masking mapper It consists of three MLP layers and uses the GeLU activation function; Step 5.2 specifically includes: setting the batch size to 64, using the Adam optimizer, and setting the initial learning rate of the network to 1. 10 -5 The weights were decayed to 0.01, and Dropout with a probability of 50% was used for regularization. The training lasted for a total of 60 rounds.

Images