A text attention-based cross-modal representation model construction method and retrieval method

Abstract

The application relates to a text attention-based cross-modal representation model construction method and a retrieval method, which adds two innovative modules in an online model: an attention-guided mask modeling module and a text enhancement module. The attention-guided mask modeling module takes the attention in the text encoding forward process as a reference to adaptively mask the words in the text description. In this way, the mask language modeling module can capture and align the semantics associated with the masked words during the prediction of the masked words, thereby more efficiently aligning the cross-modal representation. The text enhancement module expands the text description in the downstream task dataset in a lightweight manner, solves the problems of single word use and errors in the text description in the dataset, and improves the retrieval accuracy.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval, specifically to a method for constructing a cross-modal representation model based on text attention and a retrieval method. Background Technology

[0002] Cross-modal retrieval tasks (such as text-to-image retrieval and text-to-image person re-identification) refer to retrieving targets of interest from a dataset of interest in another modality using data from one modality as the query. In recent years, most downstream cross-modal retrieval tasks have been based on large-scale visual-language pre-trained (VLP) models. How to better align cross-modal feature representations and prevent VLP models from overfitting on downstream tasks has become a major research goal in this field. Current mainstream methods employ Masked Language Modeling (MLM) modules when applying VLP to downstream retrieval tasks. This module replaces some words in the text with uniform [MASK] markers, using text and image information to jointly predict the original words. In this way, the masked words act as a pivot, thereby better aligning the semantic relationships between the text context and the image. This method has achieved good results on many downstream tasks.

[0003] Existing multimodal representation learning frameworks suffer from two main problems. First, mainstream methods employ random masking strategies when using the MLM module for cross-modal representation alignment. This means that every word in a text description has an equal probability of being selected for masking. However, some meaningless words, such as 'with' and 'the', have no actual semantic correspondence in the image and its text context. Therefore, if these words are masked, the MLM module's prediction of these words does not help with cross-modal semantic feature alignment and may even diminish the original powerful representational capabilities of VLP to some extent. Secondly, compared to the training dataset used by VLP models (for example, the CLIP model has 400 million text-image pairs), downstream cross-modal retrieval datasets are typically very small (for example, the CUHK-PEDES text-image person re-identification dataset has only about 80,000 pairs). Furthermore, these small downstream datasets often suffer from inaccurate text descriptions and limited vocabulary due to human factors. Training on such small and noisy downstream datasets easily leads to overfitting of the VLP model. Summary of the Invention

[0004] The purpose of this invention is to provide a method for constructing and retrieving cross-modal representation models based on text attention, in order to solve the problems of VLP models losing strong representation capabilities on smaller downstream datasets and the use of limited and erroneous words in the text descriptions of the datasets, thereby improving retrieval accuracy.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for constructing a cross-modal representation model based on text attention is disclosed. The cross-modal representation model framework adopted by the method consists of an online model and a momentum model with the same model architecture. The momentum model is a smooth and stable version of the online model. The architecture of both the online model and the momentum model includes an image encoder, a text encoder, and a cross-modal encoder. The online model uses an image-text contrast learning module, an image-text matching learning module, and a mask language modeling module, and adds an attention-guided mask modeling module and a text enhancement module for training.

[0007] Specifically, the method updates the momentum model using an exponential moving average of the online model parameters, and then uses the features encoded by the momentum model as pseudo-labels to guide the online model to learn better representations; the learning and training process of the online model involves the following steps:

[0008] Step 1: For image-text pairs in a given dataset , to image and text The images are input into the image encoder and text encoder respectively to obtain the image encoded features. Text encoding features And attention weights in the text encoding process, where, and These are the number of image blocks and the number of text words, respectively. and As a global feature representation;

[0009] Step 2: The image-text comparison learning module learns from image encoding features. and text encoding features Calculate the contrastive learning loss between the image and the text, i.e., the ITC loss;

[0010] Step 3: Input the text and the attention weights obtained in the text encoding process in Step 1 into the attention-guided mask modeling module. The attention-guided mask modeling module selects words in the text T according to the attention weights and replaces them with [mask] flags to obtain the text with [mask] flags replaced. Input the text with [mask] flags replaced into the text encoder to obtain the text encoding features with [mask] flags replaced.

[0011] Step 4: The text encoding features and image encoding features obtained in Step 1 are fed into the cross-modal encoder. The text encoding features are used as the query vector, and the image encoding features are used as the key vector and value vector. The encoding vector of the last layer of the cross-modal encoder is used as the output feature of the cross-modal encoder. The output feature of the cross-modal encoder is input into the image-text matching learning module to calculate the ITM loss.

[0012] Step 5: Input the image encoding features obtained in Step 1 and the text encoding features with [mask] tags replaced in Step 3 into the cross-modal encoder. The text encoding features with [mask] tags replaced are used as the query vector, and the image encoding features are used as the key vector and value vector. The encoding vector of the last layer of the cross-modal encoder is used as the output of the cross-modal encoder. Input the output features of the cross-modal encoder into the attention-guided mask modeling module. The attention-guided mask modeling module feeds the text encoding features with [mask] tags replaced into a fully connected layer. The output is the probability logit of the word at that position belonging to any word in the vocabulary. Calculate the MLM loss based on the logit and the original word before it was replaced with the [mask] tag.

[0013] Step 6: Input the logit value obtained in Step 5 into the text enhancement module. The text enhancement module replaces the words marked with [mask] in the original description with words with higher logit values ​​according to the logit value. After performing the above replacement operation on all words marked with [mask] in the text, a new text is obtained. Then, the new text is used as a positive sample of the corresponding image.

[0014] Step 7: Use the sum of ITC, ITM, and MLM losses as the total loss, and use the result of the momentum model encoding as a pseudo-label to guide the online model update; specifically, the total loss gradient backpropagation updates the text encoder, image encoder, and cross-modal encoder of the online model, while the momentum model is updated according to the online model and does not need to be updated through gradient backpropagation.

[0015] Step 8: Iterate through the image-text pairs in the dataset and repeat steps 1-7 to train the online model until the preset number of training iterations or the training termination condition is reached. After training, the image encoder and text encoder in the online model together constitute a cross-modal representation model.

[0016] In step 2, the ITC loss is calculated as follows:

[0017]

[0018] in, and This represents the actual one-hot label, where H() represents the entropy function; For image-to-text similarity, The text-to-image similarity is calculated as follows:

[0019]

[0020] in, Indicates temperature parameter, This represents the similarity function.

[0021] In step 3, the attention-guided mask modeling module selects words in text T based on attention weights and replaces them with the [mask] flag, as follows:

[0022] For text The attention weights labeled [CLS] in each layer are represented as follows:

[0023]

[0024] Among them, A CLS This represents the set of attention weights corresponding to the [CLS] markers across all layers of the text encoder. This refers to the attention weights corresponding to the [CLS] markers in layer i. It refers to the number of layers in the text encoder;

[0025] The attention-guided mask modeling module uses an exponential moving average to combine the attention weights of different layers of the text encoder:

[0026]

[0027] in, For setting value, , The table refers to the attention weights of the [CLS] markers in the k-th layer text encoder. , , , , , Through Weighted average yields intermediate results. The final [CLS] label corresponds to the attention weight;

[0028] The attention weights are then subjected to a softmax operation again, as shown below:

[0029]

[0030] in, The attention weights corresponding to the i words, It is a temperature parameter used to dynamically adjust the range and interval of the softmax results. This is the total number of inputs;

[0031] The value after the softmax operation is used as a reference to mask the corresponding words. Specifically, the probability of masking the i-th word is expressed as:

[0032]

[0033] in, It's the offset. It is the amplitude;

[0034] Finally, based on the obtained probabilities, the selected words are replaced with special [mask] tags.

[0035] In step 4, the ITM loss is calculated as follows:

[0036]

[0037] Among them, y itm This represents the actual label; a value of 1 indicates a corresponding text image, while a value of 0 indicates a different text image. itm H() represents the probability that the model outputs whether the text and the image correspond. The ultimate goal of optimization is to make the model output close to 1 for positive sample pairs and close to 0 for negative sample pairs. H() represents the entropy function.

[0038] In step 5, the MLM loss is calculated as follows:

[0039]

[0040] Among them, y msk p represents the actual label of the original word corresponding to the word that was replaced with the [mask] tag. msk This represents the probability of the model output corresponding to the real word. This indicates the text that has had the [mask] tag replaced.

[0041] An instance retrieval method based on a cross-modal representation model, the retrieval method comprising the following steps:

[0042] Step 1: First, input all the images in the image library into the image encoder of the cross-modal representation model for encoding to obtain image representations, and then store all image representations into the vector database.

[0043] Step 2: Input the natural language description of the target image of interest into the text encoder of the cross-modal representation model as a query to obtain the text representation;

[0044] Step 3: Calculate the distance between the text representation obtained in Step 2 and the image representation in the vector database, and output the N images that are most similar to the text description;

[0045] The cross-modal representation model is obtained using a text attention-based cross-modal representation model construction method as described above.

[0046] By adopting the above scheme, this invention adds two highly innovative modules to the online model: an attention-guided masking modeling module and a text enhancement module. The attention-guided masking modeling module uses the attention in the text encoding forward process as a reference to adaptively mask words in the text description. In this way, words with practical meaning are more likely to be selected for masking than words without practical meaning. Thus, the masking language modeling module can capture and align the semantics associated with the masked words during prediction, thereby more efficiently aligning cross-modal representations and preventing the loss of VLP's powerful semantic understanding capabilities. The text enhancement module expands the text descriptions in the downstream task dataset in a lightweight manner. Specifically, in the process of predicting the original masked words, the masked language modeling module generates a probability value for each word in the vocabulary. Thanks to VLP's powerful semantic understanding capabilities and the information seen in the image during the prediction process, words with high probability values ​​can well describe the content in the original image, and even better match the corresponding semantics in the image. Therefore, the text enhancement module takes the words corresponding to the high probability values ​​to replace the masked words and uses them as new text descriptions for the image, thereby solving the problems of monotonous word usage and errors in the text descriptions in the dataset, and improving the retrieval accuracy. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the principle of the present invention;

[0048] Figure 2 This is a comparison chart of the ordinary MLM method and the AGM method;

[0049] Figure 3 A visualization of the cross-modal attention results;

[0050] Figure 4 A schematic diagram illustrating the text enhancement module processing;

[0051] Figure 5 This is a flowchart of the instance retrieval process. Detailed Implementation

[0052] like Figure 1As shown, this invention discloses a method for constructing a cross-modal representation model based on text attention. The cross-modal representation model framework employed in this method consists of an online model and a momentum model with the same model architecture. The momentum model is a smoothed and stable version of the online model. Both the online model and the momentum model include an image encoder, a text encoder, and a cross-modal encoder, but their learning and optimization processes differ. The momentum model uses a base model (including an image-text contrast learning module (ITC), an image-text matching learning module (ITM), and a masked language modeling module (MLM)) and updates it using the exponential moving average (EMA) of the online model parameters. The online model, on the other hand, utilizes the image-text contrast learning module (ITC), the image-text matching learning module (ITM), and the masked language modeling module (MLM), and adds an attention-guided masked modeling module (AGM) and a text enhancement module (TEM) for training.

[0053] Based on the above cross-modal representation model framework, the cross-modal representation model construction method of this invention is as follows: The momentum model is updated using the exponential moving average (EMA) of the online model parameters, and then the features encoded by the momentum model are used as pseudo-labels to guide the online model to better learn representations. The learning and training process of the momentum model is the same as existing methods, and will not be elaborated upon here. The steps of the online model's learning and training process are as follows:

[0054] Step 1: Given an image-text pair , to image and text The images are input into the image encoder and text encoder respectively to obtain the image encoded features. Text encoding features And attention weights in the text encoding process, where, and These are the number of image blocks and the number of text words, respectively. and As a global feature representation.

[0055] Step 2: The Image-Text Comparison Learning (ITC) module learns from image coding features. and text encoding features Calculate the contrastive learning loss between the image and the text.

[0056] Specifically, the similarity between text and image after softmax normalization is calculated as follows:

[0057]

[0058] in, For image-to-text similarity, For text-to-image similarity, Indicates temperature parameter, This represents the similarity function, and M represents the size of a mini-batch during training. In deep learning, model training involves dividing the dataset into batches for training.

[0059] The overall ITC loss is expressed as follows:

[0060]

[0061] in, and H represents the true one-hot label, and H() represents the entropy function.

[0062] In step 3, the attention weights from the text encoding process obtained in step 1 are input into the attention-guided mask modeling module (AGM). The attention-guided mask modeling module selects words in the text T according to the attention weights and replaces them with [mask] markers to obtain text with [mask] markers replaced. The text with [mask] markers replaced is input into the text encoder to obtain the text encoding features with [mask] markers replaced.

[0063] (1) Limitations of Ordinary MLM. Masked Language Modeling (MLM) aims to predict a masked subset of input tokens based on textual context. Ordinary MLM employs a random masking strategy, typically setting each word to have a 15% probability of being selected. Of the selected words, 80% are replaced with the [MASK] token, 10% are randomly replaced with other words, and 10% remain unchanged. MLM operates within a cross-modal interactive encoder. A cross-modal interactive encoder consists of a multi-head self-attention layer (MHSA), a multi-head cross-attention layer (MHCA), and a feedforward network (FFN). Since MHSA and FFN do not include interaction between the two modalities, only MHCA is considered here. Let... To indicate text masking, the MHCA process is as follows:

[0064]

[0065] in, It refers to the first The query matrix generated from masked text in the layer cross-modal encoder. and It refers to the first The key-value matrix generated by the image tokens in the layered cross-modal encoder. After this, the features corresponding to the [MASK] tokens in the final layer's encoded features are fed into the MLM classifier head to predict the original words. Using these masked tokens as a benchmark, semantic information related to these masked tokens is captured from the text context through an attention mechanism, achieving cross-modal representation alignment. For example, as... Figure 3 As shown, when the masked word is "heels," it is necessary to capture the "white chunk" in the text and the relevant semantics in the image in order to predict the original word "heels." However, ordinary cross-modal MLM methods all use random masking strategies, which results in a large portion of the masked words being meaningless words, such as "the," "also," and "with." These words have no relevant semantics in the text context and the image, making it impossible to align cross-modal feature representations. A cross-attention visualization between the masked words and image chunks can reflect this problem. Figure 3 As shown, when the masking word is "with", no valuable visual semantic information is captured to help align cross-modal feature representations.

[0066] (2) Attention-guided mask modeling. For example... Figure 2 As shown, the Attention-Guided Masking Modeling Module (AGM) is designed to better align cross-modal representations. While ordinary MLM methods use the same mask probability for all words, the AGM employs dynamic probabilities to adaptively mask words. It is well known that during text encoding, the features corresponding to the [CLS] markers are used as global feature representations, which are the weighted sum of all words. Therefore, the corresponding attention weights can reflect the importance of each word, indirectly indicating whether the word has rich information, and thus can serve as a reference for the probability of that word being masked. Therefore, the AGM uses the attention weights corresponding to the [CLS] markers of the text as a reference to dynamically adjust the probability of words being masked. Specifically, for text... The attention weights labeled [CLS] in each layer can be represented as:

[0067]

[0068] Among them, A CLS This represents the set of attention weights corresponding to the [CLS] markers across all layers of the text encoder. This refers to the attention weights corresponding to the [CLS] markers in layer i. It represents the number of layers in the text encoder.

[0069] The most intuitive strategy is to average the attention weights across each layer. However, as the network deepens, noise decreases, allowing the network to focus more on semantically rich words; that is, deeper attention weights are more valuable than shallower ones. Therefore, the attention-guided masking modeling module uses an exponential moving average to jointly combine the attention weights of different layers in the text encoder.

[0070]

[0071] In the final design, it is set , The table refers to the attention weights of the [CLS] markers in the k-th layer text encoder. , , , , , Through Weighted average yields intermediate results. This refers to the attention weights corresponding to the final [CLS] label. It's worth noting that... The self-attention weights include the [CLS] token, but the [CLS] token does not correspond to any actual word. Therefore, the attention weights need to be re-softmaxed to ignore the self-attention effect of the class token. This operation can be represented as follows:

[0072]

[0073] in, The attention weights corresponding to the i words, This is a temperature parameter used to dynamically adjust the range and interval of the softmax results; in this embodiment, it is set to 0.02. This represents the total number of input words (number of words + one [CLS] flag). The softmax value is used as a reference to mask corresponding words. Specifically, the probability of masking the i-th word is expressed as:

[0074]

[0075] in, It's the offset. Both the amplitude and the range can be set to control the proportion of words that are blocked in the entire text.

[0076] Finally, words are selected and replaced with special [mask] markers based on the obtained probabilities. Specifically, for each word, a random number between 0 and 1 is generated. If the random number is less than the obtained probability value, the word is replaced with a special [mask] marker.

[0077] Step 4: The text-encoded features and image-encoded features obtained in Step 1 are fed into the cross-modal encoder. The text-encoded features serve as the query vector, and the image-encoded features serve as the key and value vectors. The encoding vector of the last layer of the cross-modal encoder (i.e., the output of the cross-modal encoder) is represented as follows: ,in, It refers to the encoded features corresponding to the [CLS] marker after passing through the cross-modal encoder. This represents the encoded features of each word after it has passed through a cross-modal encoder.

[0078] The features output by the cross-modal encoder are fed into the ITM module to calculate the ITM loss and perform gradient backpropagation to train the model. The ITM loss calculation formula is shown in formula (3) below.

[0079]

[0080] Wherein, ITM loss is the text-image pair matching loss, y itm p represents the true label (1 if the text and image correspond, 0 if they do not). itm This represents the probability that the model outputs whether the text and the image correspond. The ultimate goal of optimization is to make the model's output for positive sample pairs close to 1 and for negative sample pairs close to 0.

[0081] Step 5: Input the image encoding features obtained in Step 1 and the text encoding features with [mask] tags replaced obtained in Step 3 into the cross-modal encoder. The text encoding features with [mask] tags replaced serve as the query vector, and the image encoding features serve as the key and value vectors. The encoding vector of the last layer of the cross-modal encoder (i.e., the output of the cross-modal encoder) is represented as follows: ,in, It refers to the encoded features corresponding to the [CLS] marker after passing through the cross-modal encoder. This represents the encoded features of each word (some of which are replaced with [mask] tags) after passing through the cross-modal encoder.

[0082] The features output by the cross-modal encoder are input into the MLM module to calculate the MLM loss. Specifically, the MLM module feeds the encoded features that have been replaced with the [mask] tag into a fully connected layer, and the output is the probability (logit) that the word at that position belongs to any word in the vocabulary. The MLM loss is calculated based on the logit and the standard answer (the original word before being replaced with the [mask] tag). The MLM loss is shown in the following formula (4).

[0083]

[0084] Among them, y msk p represents the actual label of the original word corresponding to the word that was replaced with the [mask] tag. msk This represents the probability of the model output corresponding to the real word. This represents the text whose [mask] tag has been replaced (this is the input to the model).

[0085] Step 6: Input the probability (logit) of the words replaced with [mask] tags obtained in Step 5 belonging to any word in the vocabulary into the text augmentation module (TEM). The text augmentation module replaces the words replaced with [mask] tags in the original description with words with higher logit values ​​based on the logit values ​​(logit value reflects the similarity to the original replaced word; the higher the logit value, the closer the meaning is to the original word). After performing the above replacement operation on all words replaced with [mask] tags in the text, a new text is obtained. This new text will be used as a positive sample of the corresponding image (meaning it can also appropriately describe the image) to train the model.

[0086] Specifically, inaccurate descriptions are inevitable in textual descriptions. For example... Figure 4 As shown, jeans described as "black" in the original description are actually more appropriately described as "blue." On the other hand, the vocabulary used to describe the same object is rather limited; for example, "shoes" is more commonly used, while "sneakers" is rarely seen in descriptions. Due to limitations in data scale and various human factors leading to limited and incorrect vocabulary in text descriptions, VLP is prone to overfitting when trained on the downstream TBPS dataset, thus losing its powerful semantic understanding capabilities. To address this issue, building upon the AGM's ability to select more informative words for masking, we designed a Text Augmentation Module (TEM) to provide more diverse and high-quality text descriptions. Since VLP possesses strong semantic understanding capabilities, the logit value of the MLM classification head can reflect the semantic similarity between the predicted word and the original masked word. Furthermore, thanks to MLM's utilization of image information when predicting the original word, these words with higher logit values ​​can even more accurately describe the corresponding content in the image. For example... Figure 4 As shown, the Top-5 predicted word list and the original masked words are presented. It's clear that these words all seem to be suitable for describing the content in the image, or even better; for example, "dark" might be a better description of the color of the pants than the original "black". TEM uses... The probability of replacing masked words with words with higher logit values ​​is determined, aiming to enrich the original text description and reduce noise without consuming any additional resources. For masked text containing masked words... and its corresponding predicted logit value A sorted list can be constructed:

[0087]

[0088] in, This refers to the softmax function. This represents the top-m truncation function, which selects the m words with the highest logit. This yields the masked text. Given a potential replacement list, a word is randomly sampled from it, ensuring that the sampled word is different from the original word. This process can be formalized as follows:

[0089]

[0090] Among them, W i The term "replacement" refers to the specific word to be replaced. "Multinomial" represents the multinomial distribution sampling function, which is the normalized logit value obtained from the previous formula. The above formula indicates that the replacement word needs to be sampled according to the multinomial distribution.

[0091] Then, the original words are replaced with new words to obtain a new text description. Replacing meaningful words with synonyms not only preserves the original information, making the text description richer, but also makes it more accurate for the corresponding image. This is because the prediction of the original words uses not only textual context information but also image information. The resulting new description will be used as a positive sample for representation learning of the corresponding image.

[0092] Step 7: Online Model Update In addition to the ITC, ITM, and MLM losses mentioned above, the momentum model encoding result will be used as a pseudo-label to guide the online model update. This is consistent with the base model (ALBEF) used in this patent and will not be elaborated further. Therefore, the total loss of the final trained model is:

[0093]

[0094] The total loss gradient backpropagation updates the text encoder, image encoder, and cross-modal encoder. The momentum model is updated based on the online model and does not require updating via gradient backpropagation.

[0095] Step 8: Iterate through the image-text pairs in the dataset and repeat steps 1-7 to train the online model until the preset number of training iterations or the training termination condition is reached. After training, the image encoder and text encoder in the online model together constitute a cross-modal representation model.

[0096] After constructing the cross-modal representation model using the above method, the model is directly applied to the instance retrieval system. The input to the instance retrieval system is a text description, and the output is the N images in the image database that are most similar to the text description. The specific retrieval method is as follows: Figure 5 As shown:

[0097] Step 1: First, input all the images in the image library into the image encoder of the cross-modal representation model for encoding to obtain image representations, and then store all image representations into the vector database.

[0098] Step 2: Input the natural language description of the target image of interest into the text encoder of the cross-modal representation model as a query to obtain the text representation.

[0099] Step 3: Calculate the distance between the text representation obtained in Step 2 and the image representation in the vector database, and output the N images that are most similar to the text description.

[0100] The method of this invention has achieved state-of-the-art performance on three mainstream text image person re-identification datasets (CUHK-PEDES, ICFG-PEDES, RSTPReid).

[0101] The method of this invention includes two highly innovative modules: an attention-guided mask modeling module and a text enhancement module.

[0102] The attention-guided masking modeling module references the attention during the text encoding feedforward process to adaptively mask words in the text description. In this way, meaningful words are more likely to be selected for masking than meaningless words (because meaningful words have higher attention weights). This allows the masking language modeling module to capture and align the semantics associated with the masked words during prediction, thus more efficiently aligning cross-modal representations and preventing the loss of VLP's powerful semantic understanding capabilities. For example, on the CUHK-PEDES dataset, adding this module improved the R@1 accuracy by 1.07%.

[0103] The text augmentation module expands the text descriptions in the downstream task dataset in a lightweight manner. Specifically, the masked language modeling module generates a probability value for each word in the vocabulary during the prediction of the original masked word. Thanks to VLP's powerful semantic understanding and the image information seen during the prediction process, words with high probability values ​​can well describe the content of the original image, and even better match the corresponding semantics in the image. Therefore, the text augmentation module replaces the masked word with the word corresponding to the high probability value as the new text description of the image, thus solving the problems of monotonous vocabulary and errors in the text descriptions in the dataset. After adding the text augmentation module, the R@1 accuracy on the CUHK-PEDES dataset improved by 0.36%. When both modules work together, the R@1 accuracy on the CUHK-PEDES dataset improved by 1.85%.

[0104] The present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist alone and not assembled into the electronic device.

[0105] The aforementioned computer-readable medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0106] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0107] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this disclosure.

[0108] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0109] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A text attention-based cross-modal representation model construction method, characterized in that: The method adopts a cross-modal representation model framework composed of an online model and a momentum model with the same model architecture, and the momentum model is a smooth and stable version of the online model; the architecture of the online model and the momentum model both includes an image encoder, a text encoder and a cross-modal encoder; the online model uses an image-text contrast learning module, an image-text matching learning module and a mask language modeling module, and adds an attention-guided mask modeling module and a text enhancement module for training; Specifically, the method updates the momentum model through the exponential moving average of the online model parameters, and then uses the features encoded by the momentum model as pseudo labels to guide the online model to perform better representation learning; wherein the learning and training process of the online model is as follows: Step 1, for image-text pairs in a given dataset , input the image and text into image encoder and text encoder respectively, obtain image encoding features , text encoding features and attention weights in text encoding process, where, and are the number of image patches and text words respectively, and are global feature representations; Step 2: The image-text comparison learning module learns from image encoding features. and text encoding features Calculate the contrastive learning loss between the image and the text, i.e., the ITC loss; Step 3, input the attention weight in the text encoding process obtained in step 1 and the text into the attention-guided mask modeling module, the attention-guided mask modeling module selects and replaces the [mask] mark according to the attention weight in the text T, and obtains the text with the [mask] mark replaced; input the text with the [mask] mark replaced into the text encoder to obtain the text encoding features with the [mask] mark replaced; Step 4, input the text encoding features and the image encoding features obtained in step 1 into the cross-modal encoder, the text encoding features as the query vector, the image encoding features as the key vector and the value vector, and the encoding vector of the last layer of the cross-modal encoder as the output features of the cross-modal encoder; input the output features of the cross-modal encoder into the image-text matching learning module to calculate the ITM loss; Step 5, input the image encoding features obtained in step 1 and the text encoding features with the [mask] mark replaced obtained in step 3 into the cross-modal encoder, the text encoding features with the [mask] mark replaced as the query vector, and the image encoding features as the key vector and the value vector, and the encoding vector of the last layer of the cross-modal encoder as the output of the cross-modal encoder; input the output features of the cross-modal encoder into the attention-guided mask modeling module, and the attention-guided mask modeling module inputs the text encoding features with the [mask] mark replaced into a fully connected layer, and the output is the probability size logit of the word at this position belonging to any word in the word library, and the MLM loss is calculated according to the logit and the original word replaced by the [mask] mark; Step 6, input the logit value obtained in step 5 into the text enhancement module, and replace the words replaced by the [mask] mark in the original description with a certain probability according to the logit value, and obtain a new text after the above replacement operation is performed on all the words replaced by the [mask] mark in the text, then the new text is used as the positive sample corresponding to the image; Step 7, using the sum of ITC, ITM and MLM losses as the total loss, and the result of the momentum model coding as pseudo-label to guide the online model update; specifically, the total loss gradient backpropagation updates the text encoder, image encoder and cross-modal encoder of the online model, and the momentum model is updated according to the online model, without the need for gradient backpropagation to update; Step 8, iterate the image-text pairs in the dataset, and repeat steps 1-7 to learn and train the online model until the preset training times or the training end condition is reached, and after the training ends, the image encoder and text encoder in the online model jointly constitute the cross-modal representation model; In step 2, the ITC loss is calculated as follows: where, and represents the true one-hot label, H() represents the entropy function; is the similarity of image to text, is the similarity of text to image, which is calculated as follows: wherein denotes a temperature parameter, denotes a similarity function; In step 3, the attention-guided mask modeling module selects and replaces the words in the text T with the [mask] mark according to the attention weight as follows: For text denote the attention weights of the [CLS] token at each layer as: wherein A CLS a set of attention weights corresponding to the [CLS] token representing all layers of the text encoder, refers to an attention weight corresponding to the [CLS] token in the i-th layer, is the number of layers in the text encoder; The attention-guided mask modeling module uses an exponential moving average to combine the attention weights of different layers of the text encoder: wherein, is a set value, , Table refers to the attention weight of the [CLS] token in the k-th layer text encoder, , , , , , is obtained by weighting the intermediate result, is the final attention weight corresponding to the [CLS] token; The attention weight is re-softmaxed as follows: wherein, a corresponding attention weight for i words, is a temperature parameter to dynamically adjust the range and interval of the softmax result, is the total number of inputs; The value after the softmax operation is used as a reference to mask the corresponding words, specifically, the probability of masking the i-th word is represented as: wherein is an offset, is an amplitude; Finally, according to the obtained probability, the word is selected and replaced with the special [mask] mark; In step 4, the ITM loss is calculated as follows: where y itm represents the true label, which is 1 if the text image is corresponding, and 0 if the text image is not corresponding; p itm represents the probability of the model outputting whether the text and image are corresponding; the final optimization goal is to make the model output close to 1 for the positive sample pair and close to 0 for the negative sample pair; H() represents the entropy function; In step 5, the MLM loss is calculated as follows: where y msk represents the true label of the original word corresponding to the word replaced by the [mask] token, p msk represents the probability output by the model corresponding to the true word, represents the text in which the [mask] token is replaced.

2. An instance retrieval method based on a cross-modal representation model, characterized in that: The retrieval method comprises the following steps: Step 1, first input all the pictures in the picture library into the image encoder of the cross-modal representation model for encoding to obtain picture representations, and store all the picture representations in a vector database; Step 2, input the natural language description of the target image of interest into the text encoder of the cross-modal representation model as a query to obtain a text representation; Step 3, calculate the distance between the text representation obtained in step 2 and the picture representations in the vector database, and output the N pictures most similar to the text description; The cross-modal representation model is obtained by using the cross-modal representation model construction method based on text attention according to claim 1.

Images