Combined image retrieval method based on picture mapper and text self-covering mapper

Abstract

A combined image retrieval method based on a picture mapper and a text self-covering mapper comprises the following steps: firstly, carrying out operations such as zooming, noise adding and the like on an input picture, then inputting the picture into a frozen pre-training image encoder to obtain an image feature # imgabs0 #, and converting the image feature # imgabs1 # into a pseudo word mark s1 by using the image mapper; and converting the image feature # imgabs2 into a pseudo word mark s2 by using a text self-covering mapper, respectively replacing a mark symbol in a pre-provided prompt text with the pseudo word mark s2, and splicing the input modified text to the prompt text to obtain two new texts x1 and x2. And inputting the text x1 and the text x2 into the frozen pre-training text encoder to obtain two features, performing mean value fusion on the two features, and finally comparing the obtained mean value feature with a target image. According to the method, the defect that an expensive triple is used for training a combined image retrieval model is overcome, and the generalization ability of the model to a combined image retrieval downstream task is also improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal retrieval based on text and image, and in particular to a method using a picture-based mapper and a text self-masking mapper. Background Art

[0002] The combined image retrieval task based on text and images aims to find the target image that matches the original image and the edited text for the image. This method provides higher freedom and better user experience for image-based search engine applications (such as e-commerce platforms and Internet searches), because language is the most natural way for human-computer interaction. Compared with the traditional method of searching for target images using only text or searching for similar images using images, combined image retrieval combines the information of text and images, and can retrieve target images more accurately in most cases.

[0003] In the combined image retrieval task, most traditional models rely on post-image text feature fusion, that is, combining visual and language features after they are extracted using different encoders. Existing visual language base models (such as CLIP, ALIGN, etc.) pre-train image and text encoders on hundreds of millions of large-scale image-text pairs. Because these models have been trained on a large amount of data, they have seen different texts used to describe a certain picture, so they can handle different tasks. Among them, the pre-trained CLIP model performs well. Some methods use the pre-trained BERT model to fuse image and text features, and others train a shallow transformer from scratch to directly fuse image and language features at input. However, these methods ultimately need to use specific datasets for combined image retrieval tasks to adjust their models, so as to fully utilize the combination capabilities of pre-trained language models.

[0004] The dataset for combined image retrieval tasks mainly consists of triplets of original images, query text, and target images. Unlike traditional image-text matching datasets, this type of triple data is almost impossible to collect through web crawlers, and expensive manpower is required to manually create each triple. Therefore, existing triple datasets are small in scale and are all targeted at a small field. In order to improve the scalability and practicality of combined image retrieval tasks, it is very important to optimize the training data and training mode under traditional models. Summary of the invention

[0005] In order to overcome the problem of small application scope and poor scalability of combined image retrieval tasks under the current existing dataset training, this paper proposes a combined image retrieval method based on image mapper and text self-masking mapper to solve the generalization problem of combined image retrieval. This method does not require expensive triplet training datasets, only image captions and unlabeled image datasets, and its effect is similar to the performance of the method trained with labeled triplet data.

[0006] In order to achieve the above object, a combined image retrieval method based on an image mapper and a text self-masking mapper of the present invention adopts the following technical solution:

[0007] Step 1: Read the dataset and use the pictures and text descriptions as the data input for the model.

[0008] Step 1.1: First, scale the image to a predetermined size, and perform data enhancement by horizontal flipping, randomly adding Gaussian noise, etc.

[0009] Step 1.2, use the token extractor in the natural language tool to extract keywords from the current text, add the extracted keywords to a list and remove duplicates to obtain a keyword list, which records all the keywords contained in the current text.

[0010] Step 1.3, traverse the keyword list, replace the keywords in the text with the '[$]' symbol, and obtain a new text description.

[0011] Step 2: Train the image mapper f M , converting image features into pseudo-language token embeddings: Where s is a pseudo-language tag embedding, is the image feature.

[0012] Step 2.1: Pass the input image through the frozen visual encoder f I Processing to obtain image features

[0013] Step 2.2, construct an image mapper f with parameters M M .

[0014] Step 2.3, using the image mapper f M The image features Converted into pseudo-word token s.

[0015] Step 2.4, construct the text prompt: "A photo of", and then splice the image features The converted pseudo-word tag s obtains the concatenated text “A photo of[s]”.

[0016] Step 2.5, stitch the text Input frozen text encoder f φ Get the concatenated text features

[0017] Step 2.6, the training purpose is to obtain the concatenated text features Can fully represent the input image information, that is, train the image mapper f M The ability to map image features to pseudo-word tokens. To achieve this, the contrast loss relative to the mapper is minimized so that similar images and text are closer in the same embedding space, and dissimilar ones are farther apart;

[0018] The functional expression of contrast loss is as follows:

[0019]

[0020] in, represents the contrast loss from text to image, represents the contrastive loss from image to text. τ is a temperature parameter used to scale the similarity score. represents the number of samples in a training batch, p i Yes Normalize to get .

[0021] Step 3: Since the input of the text self-masking mapper φ is plain text data, and the present invention uses it for the combined image retrieval task, the method of adding noise is adopted to bridge the modality gap between plain text and pictures.

[0022] Step 3.1: Enter the text x c The frozen pre-trained text encoder f φ Get text feature z c .

[0023] Step 3.2, for text feature z c Add noise n.

[0024] Step 4: Train the text self-masking mapper φ to capture the information in the image through pure text training.

[0025] Step 4.1, construct a text self-masking mapper φ with parameter N.

[0026] Step 4.2: Use the text self-masking mapper φ to map the text feature z added with noise in step 3.2 c Convert to pseudo-word tokens

[0027] Step 4.3, using the text description corresponding to the text obtained in step 1.3, with the keyword replaced by the '[$]' symbol, mark the pseudo-word Replace all the '[$]' symbols and record it as

[0028] Step 4.4, Input to the frozen pre-trained text encoder f φ Get text features

[0029] Step 4.5, for the two text features z c and Minimize the mean square error loss between the two to train the text self-masking mapping;

[0030] The mean square error loss function expression is as follows:

[0031]

[0032] Step 5: According to the two loss functions in step 2 and step 4, the model is optimized using the back propagation algorithm and the gradient descent algorithm; specifically, the following steps are performed:

[0033] Step 5.1, based on the actual input and expected output, the overall error formula is obtained:

[0034]

[0035] In the formula It is the text feature after calculation of splicing With image features The contrast loss function is is to calculate the text feature z c and the text features after being replaced by pseudo-word tokens The mean square error loss function is the difference between .

[0036] Step 5.2: Use the back propagation algorithm and gradient descent algorithm to optimize the model parameters; set the batch size to 64, use the Adam optimizer, and set the initial learning rate of the network to 1*10 -5 , the weight decay is 0.01, and Dropout with probability 50% is used for regularization, and a total of 60 rounds of training are performed.

[0037] Step 6: Test the model.

[0038] In step 6.1, provide a prompt text, such as "a photo of[$]". The images in the dataset are passed through the frozen pre-trained visual encoder, and the features obtained are used as the query set.

[0039] Step 6.2: First, extract features from the image in the input data through the frozen pre-trained visual encoder. Will Input to the image mapper f M The pseudo-words s1 and s2 are obtained from the text self-masking mapper φ, which replace the '[$]' symbol in the pre-provided prompt text, and the text part in the data is concatenated to the prompt text to obtain two new texts x1 and x2.

[0040] Step 6.3, input the text x1, x2 into the frozen pre-trained text encoder f φ In the above example, two features are obtained and the two features are fused.

[0041] Step 6.4, calculate the cosine similarity between the fusion features obtained in step 6.3 and the features in the query set, and then sort them to obtain the corresponding query results.

[0042] Preferably, in step 1.1, the image is first scaled to a size of 224px*224px, and data enhancement is performed by performing horizontal flipping and randomly adding Gaussian noise.

[0043] Preferably, the natural language tool described in step 1.2 is the spaCy natural language processing library, and the types of extracted keywords are nouns ('NOUN' or 'PROPN' type) and adjectives ('ADJ' type).

[0044] Preferably, the replacement rule in step 1.3 is to replace all keywords in the original text with '[$]' symbols.

[0045] Preferably, the visual encoder f described in step 2.1 I ResNet network, Vision Transformer network.

[0046] Preferably, the image mapper f in step 2.2 M It consists of three layers of MLP with 512 hidden units and uses ReLU activation function, with about 800,000 parameters.

[0047] Preferably, the text encoder f described in step 2.5 φ It is a Transformer network.

[0048] Preferably, the noise added in step 3.2 is That is, a random scalar multiplied by a random vector drawn from a Gaussian distribution.

[0049] Preferably, the text self-masking mapper φ in step 4.1 is composed of three layers of MLP and uses a GeLU activation function.

[0050] Preferably, step 5.2 specifically includes: setting the batch size to 64, using the optimizer Adam, and setting the initial learning rate of the network to 1*10 -5 , the weight decay is 0.01, and Dropout with probability 50% is used for regularization, and a total of 60 rounds of training are performed.

[0051] Preferably, the fusion method in step 6.3 is mean fusion.

[0052] The working principle of the present invention is as follows: the present invention mainly solves the problems of expensive triple data sets and poor generalization of downstream tasks in combined image retrieval tasks through two mapping networks. First, the image mapper is trained using pictures and simple prompt texts so that it can reconstruct visual representations in the text embedding space. Afterwards, the text self-masking mapper is trained using text labels, which can make up for the problem of lack of text diversity during image mapper training, and only text needs to be used for training. When training these two mappers, only a large number of image-text pairs are needed as input, and expensive triple data is no longer needed. At the same time, since a large number of image-text pairs are sufficient for the model to learn knowledge in various fields, rather than being limited to a specific task, the model's generalization ability for downstream tasks is also improved.

[0053] The image mapper and text self-masking mapper proposed in the present invention can be extracted separately as an independent model for training, and can also achieve good experimental results. Compared with the image mapper, the text self-masking mapper only uses text labels during training, so its training speed alone is very fast. Compared with various complex multimodal retrieval models (time complexity is O(mn)), the present invention adopts a dual-stream feature extraction network, which only needs to use a text feature extraction network and a picture feature extraction network to extract text and picture features respectively, and the time complexity is O(m+n). Therefore, compared with the traditional model of combined image retrieval tasks, the present invention has a faster matching retrieval effect.

[0054] Advantages of the present invention: 1. It provides a new training idea for the field of combined image retrieval. 2. The training data no longer relies on the expensive triplet dataset in the combined image retrieval task, and only text titles and unlabeled image datasets can be used for training. 3. It has better generalization and is no longer targeted at a specific field, but can solve various combined image retrieval models. 4. In the zero-shot combined image retrieval task, on the dataset FashionIQ, the accuracy rank@10 increased by 4.09%, and the rank@50 increased by 4.28%. The effect is comparable to that of the model trained using the traditional triplet dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart for training the image mapper and text self-masking mapper in the method of the present invention.

[0056] Figure 2 It is the whole of combined image retrieval performed by the present invention. DETAILED DESCRIPTION

[0057] The present invention is further described below in conjunction with the accompanying drawings and implementation methods.

[0058] A combined image retrieval method based on image mapper and text self-occlusion mapping, the training process is as follows Figure 1 As shown, the specific implementation steps are as follows:

[0059] 1) Read the dataset and use each pair of matching images and image captions as the data input for the model.

[0060] 11) First scale the image to a size of 224px*224px, perform horizontal flipping, randomly add Gaussian noise, etc. for data enhancement.

[0061] 12) Use the token extractor in the natural language tool spaCy to extract keywords from the current text. The keyword types are nouns ('NOUN' or 'PROPN' type) and adjectives ('ADJ' type). Add the extracted keywords to a list and remove duplicates to obtain a keyword list, which records all the keywords contained in the current text.

[0062] 13) Traverse the keyword list and replace all keywords in the text with '[$]' symbols to obtain a new text description.

[0063] 2) If Figure 1 As shown, the image mapper f is trained M , converting image features into pseudo-language token embeddings: Where s is a pseudo-language tag embedding, is the image feature.

[0064] 21) Pass the input image through the frozen visual encoder f I Processing to obtain image features Visual encoder f I The pre-trained visual encoder Vision Transformer-base in CLIP is used, which is mainly composed of a normalized Norm layer, a multi-head self-attention mechanism layer, and a fully connected MLP layer.

[0065] 22) Build an image mapper f M , consists of a three-layer MLP with 512 hidden units and uses the ReLU activation function, with approximately 800,000 parameters.

[0066] 23) Using image mapper M The image features Converted into pseudo-word token s:

[0067] 24) Construct a text prompt: "A photo of", followed by the image features The converted pseudo-word tag s obtains the concatenated text “A photo of[s]”.

[0068] 25) Splice the text Input frozen text encoder f φ Get the concatenated text features Text Encoderf φ The pre-trained text encoder Transformer in CLIP is used. When the text encoder is Resnet, it is mainly composed of convolutional layers CNN and residual blocks.

[0069] 26) The purpose of training is to obtain the concatenated text features Can fully represent the input image information, that is, train the image mapper f M The ability to map image features to pseudo-word tokens. To achieve this, the contrast loss relative to the mapper is minimized so that similar images and text are closer in the same embedding space, and dissimilar ones are farther apart;

[0070] The functional expression of contrast loss is as follows:

[0071]

[0072] in, represents the contrast loss from text to image, represents the contrastive loss from image to text. τ is a temperature parameter used to scale the similarity score. represents the number of samples in a training batch, p i Yes Normalize to get .

[0073] 3) Since the input of the text self-masking mapper φ is plain text data, and the present invention intends to use it for the combined image retrieval task, a noise adding method is adopted to bridge the modality gap between plain text and images.

[0074] 31) Input text x c The frozen pre-trained text encoder f φ Get text feature z c .

[0075] 32) For text feature z c Adding Noise That is, a random scalar value is multiplied by a random vector drawn from a Gaussian distribution. Unif(0,1) represents a uniform distribution, which means that the scalar value is between 0 and 1, and any value has the same probability of appearing. is a normal distribution, and each component is independently drawn from a normal distribution with mean 0 and variance 1.

[0076] 4) Train a text self-masking mapper φ, and capture the information in the image by training φ with pure text.

[0077] 41) Construct a text self-masking mapper φ with parameter N, which consists of three layers of MLP and uses the GeLU activation function.

[0078] 42) Use the text self-masking mapper φ to map the text feature z added with noise in step 3.2 c Convert to pseudo-word tokens

[0079] 43) Use the text description corresponding to the text obtained in step 1.3 to replace the keyword with the '[$]' symbol, and mark the pseudo-word Replace all the '[$]' symbols and record it as

[0080] 44) Input to the frozen pre-trained text encoder f φ Get text features

[0081] 45) For the two obtained text features z c and Minimize the mean square error loss between the two to train the text self-masking mapper φ;

[0082] The mean square error loss function expression is as follows:

[0083]

[0084] 5) According to the two loss functions in step 2 and step 4, the model is optimized using the back propagation algorithm and the gradient descent algorithm.

[0085] 51) According to the actual input and expected output, the overall error formula is obtained, which is:

[0086]

[0087] In the formula It is the text feature after calculation of splicing With image features The contrast loss function is is to calculate the text feature z c and the text features after being replaced by pseudo-word tokens The mean square error loss function is the difference between .

[0088] 52) Use the back propagation algorithm and gradient descent algorithm to optimize the model parameters; set the batch size to 64, use the Adam optimizer, and set the initial learning rate of the network to 1*10 -5 , the weight decay is 0.01, and Dropout with probability 50% is used for regularization, and a total of 60 rounds of training are performed.

[0089] 6) Test the model on the FashionIQ dataset, such as Figure 2 As shown, the query data consists of two parts: a picture and a modification instruction. The invention designs a prompt text template for the data set: "a photo of[$]+modification instruction".

[0090] 61) First, the image in the input data is passed through the frozen pre-trained visual encoder to extract features Will Input to the image mapper f M The pseudo-words s1 and s2 are obtained from the text self-masking mapper φ, which replace the '[$]' symbol in the pre-provided prompt text, and the text part in the data is concatenated to the prompt text to obtain two new texts x1 and x2.

[0091] 62) Input the text x1 and x2 into the frozen pre-trained text encoder f φIn the above example, two features are obtained and the two features are averaged and fused. The cosine similarity between the obtained fused features and the features in the query set is calculated, and then they are sorted according to the similarity to obtain the top 50 images with the highest matching rate for each image-text input pair. These 50 images are then compared with the true answer to calculate the accuracy-related indicators.

[0092] 63) In order to verify the accuracy and effectiveness of the actual application of the method of the present invention. The present invention calculates the accuracy rank@10 and rank@50 values ​​on the FashionIQ dataset in detail to evaluate the algorithm performance and shows the experimental results in Table 1. According to the experimental results, the following conclusions can be drawn: (1) The accuracy is significantly improved. After using the mapping model of the invention, there is a great improvement on the FashionIQ dataset. Compared with the zero-shot combined image retrieval task, the accuracy rank@10 increased by 4.09% and the rank@50 increased by 4.28%. Compared with the model trained on the FashionIQ dataset, there is a significant improvement in the Shirt and Toptee categories, and the overall effect is the same. (2) After the model processes the image through the image mapper and the self-masking text mapper to obtain pseudo-words, it uses the text encoder to extract features for retrieval, which shows that the model of the invention can indeed allow the mapper to extract image information. Moreover, since a large number of image-text pairs are used for training, the effect is comparable to that of the model trained specifically on the FashionIQ dataset, which also demonstrates that the invented model has good generalization ability for downstream tasks in various fields.

[0093] Table 1: Retrieval results on the FashionIQ dataset

[0094]

[0095] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A combined image retrieval method based on a picture mapper and a text self-occlusion mapper, characterized in that: Training the image mapper and the text self-occlusion mapper using images and image captions involves the following steps: Step 1: Read the dataset and use the pictures and text descriptions as the data input for the model. Step 2: Train the image mapper f M , converting image features into pseudo-language token embeddings: Where s is a pseudo-language tag embedding, is the image feature; Step 3: Use the noise adding method to bridge the modality gap between plain text and images; Step 4: train the text self-occlusion mapper φ to capture the information in the image through pure text training; Step 5: According to the two loss functions in step 2 and step 4, the model is optimized using the back propagation algorithm and the gradient descent algorithm; Step 6: Test the model.

2. A combined image retrieval method based on a picture mapper and a text self-occlusion mapper as claimed in claim 1, characterized in that: Step 1 specifically includes: Step 1.1, first scale the image to a predetermined size, and perform horizontal flipping, random Gaussian noise addition, etc. for data enhancement; Step 1.2, using the token extractor in the natural language tool to extract keywords from the current text, adding the extracted keywords to a list and removing duplicates to obtain a keyword list, which records all the keywords contained in the current text; Step 1.3, traverse the keyword list, replace the keywords in the text with the '[$]' symbol, and obtain a new text description.

3. The combined image retrieval method based on a picture mapper and a text self-occlusion mapper as claimed in claim 1, characterized in that: Step 2 specifically includes: Step 2.1: Pass the input image through the frozen visual encoder f I Processing to obtain image features Step 2.2, construct an image mapper f with parameters M M ; Step 2.3, using the image mapper f M The image features Converted into pseudo-word tokens; Step 2.4, construct the text prompt: "A photo of", and then splice the image features The converted pseudo-word tag s obtains the concatenated text "A photo of[s]"; Step 2.5, stitch the text Input frozen text encoder f φ Get the concatenated text features Step 2.6, minimize the contrast loss relative to the mapper so that similar images and texts are closer in the same embedding space and dissimilar ones are farther away; The functional expression of contrast loss is as follows: in, represents the contrast loss from text to image, represents the contrastive loss from image to text, τ is a temperature parameter used to scale the similarity score, represents the number of samples in a training batch, p i Yes Normalize to get .

4. The combined image retrieval method based on a picture mapper and a text self-occluding mapper as claimed in claim 1, characterized in that: Step 3 specifically includes: Step 3.1: Enter the text x c The frozen pre-trained text encoder f φ Get text feature z c ; Step 3.2, for text feature z c Add noise n.

5. A combined image retrieval method based on a picture mapper and a text self-occlusion mapper as claimed in claim 4, characterized in that: Step 4 specifically includes: Step 4.1, construct a text self-masking mapper φ with parameter N; Step 4.2: Use the text self-masking mapper φ to map the text feature z added with noise in step 3.2 c Convert to pseudo-word tokens Step 4.3, using the text description corresponding to the text obtained in step 1.3, with the keyword replaced by the '[$]' symbol, mark the pseudo-word Replace all the '[$]' symbols and record it as Step 4.4, Input to the frozen pre-trained text encoder f φ Get text features Step 4.5, for the two text features z c and Minimize the mean square error loss between the two to train the text self-masking mapping; The mean square error loss function expression is as follows:

6. A combined image retrieval method based on a picture mapper and a text self-occluding mapper as claimed in claim 5, characterized in that: Step 5 specifically includes: Step 5.1, based on the actual input and expected output, the overall error formula is obtained: In the formula It is the text feature after calculation of splicing With image features The contrast loss function is Is to calculate the text feature z c and the text features after being replaced by pseudo-word tokens The mean square error loss function of the difference between ; Step 5.2: Use the back propagation algorithm and gradient descent algorithm to optimize the model parameters; set the batch size to 64, use the Adam optimizer, and set the initial learning rate of the network to 1*10 -5 , the weight decay is 0.01, and Dropout with probability 50% is used for regularization, and a total of 60 rounds of training are performed.

7. The combined image retrieval method based on a picture mapper and a text self-occlusion mapper as claimed in claim 1, characterized in that: Step 6 specifically includes: Step 6.1, provide prompt text, pass the images in the dataset through the frozen pre-trained visual encoder, and the features obtained are used as the query set; Step 6.2: First, extract features from the image in the input data through the frozen pre-trained visual encoder. Will Input to the image mapper f M and text from the masking mapper φ, and obtain pseudo-words s1 and s2, respectively, and replace the '[$]' symbol in the pre-provided prompt text with them, and concatenate the text part in the data to the prompt text to obtain two new texts x1 and x2; Step 6.3, input the text x1, x2 into the frozen pre-trained text encoder f φ In the above example, two features are obtained and the two features are fused; Step 6.4, calculate the cosine similarity between the fusion features obtained in step 6.3 and the features in the query set, and then sort them to obtain the corresponding query results.

8. The combined image retrieval method based on a picture mapper and a text self-occlusion mapper as claimed in claim 2, characterized in that: In step 1.1, the image is first scaled to 224px*224px, and horizontally flipped, and data enhancement is performed by randomly adding Gaussian noise; The natural language tool described in step 1.2 is the spaCy natural language processing library, and the types of extracted keywords are nouns ('NOUN' or 'PROPN' class) and adjectives ('ADJ' class); The replacement rule described in step 1.3 is to replace all the keywords in the original text with the '[$]' symbol.

9. The combined image retrieval method based on a picture mapper and a text self-occluding mapper as claimed in claim 1, characterized in that: The visual encoder f described in step 2.1 I ResNet network, Vision Transformer network; The image mapper f described in step 2.2 M It consists of three layers of MLP with 512 hidden units and uses ReLU activation function, with about 800,000 parameters; The text encoder f described in step 2.5 φ It is a Transformer network.

10. The combined image retrieval method based on a picture mapper and a text self-occlusion mapper as claimed in claim 6, characterized in that: The noise added in step 3.2 is That is, a random scalar multiplied by a random vector drawn from a Gaussian distribution. Unif(0,1) represents a uniform distribution, which means that the scalar value is between 0 and 1, and any value has the same probability of appearing. is a normal distribution, and each component is independently drawn from a normal distribution with a mean of 0 and a variance of 1; The text self-masking mapper φ in step 4.1 consists of three layers of MLP and uses the GeLU activation function; Step 5.2 specifically includes: setting the batch size to 64, using the optimizer Adam, and setting the network initial learning rate to 1*10 -5 , the weight decay is 0.01, and Dropout with probability 50% is used for regularization, and a total of 60 rounds of training are performed.

Images