Cross-modal representation model construction method and retrieval method based on text attention

Abstract

The invention relates to a text attention-based cross-modal representation model construction method and a text attention-based cross-modal representation model retrieval method. Two highly innovative modules, namely an attention guidance-based mask modeling module and a text enhancement module, are added in an online model. Wherein the attention-guided mask modeling module takes the attention in a text coding forward process as a reference and performs self-adaptive masking on words in text description, so that the mask language modeling module can capture semantics associated with masking words for alignment in a masking word prediction process; and therefore, the cross-modal characterization can be aligned more efficiently. The text enhancement module expands text description in a downstream task data set in a lightweight mode, the problems that the text description in the data set is single in word and errors exist are solved, and the retrieval accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal retrieval, and in particular to a cross-modal representation model construction method and a retrieval method based on text attention. Background Art

[0002] Cross-modal retrieval tasks (such as image and text retrieval, text and image person re-identification) refer to using data from one modality as a query to retrieve targets of interest in another modality data set. In recent years, downstream cross-modal retrieval tasks have mostly been based on large-scale visual language pre-training (VLP) models. How to better align cross-modal feature representations and prevent VLP models from overfitting on downstream tasks has become the main research goal in this field. The current mainstream method uses a masked language modeling (MLM) module when using VLP for downstream retrieval tasks. This module replaces some words in the text with a unified [MASK] sign, and uses text and image information to jointly predict the original words. In this way, the masked words are equivalent to pivots, thereby better aligning the relevant semantics between the text context and the image. This method has achieved good results on many downstream tasks.

[0003] There are two main problems with the existing multimodal representation learning framework. First, the mainstream methods use the random masking strategy when using the MLM module to perform cross-modal representation alignment, which means that each word in a text description has an equal probability of being selected for masking. However, some meaningless words such as ('with', 'the') have no actual semantic correspondence in the image and its text context. Therefore, if these words are masked, the MLM module will not help the cross-modal semantic feature alignment in the process of predicting these words, and will even lose the original powerful representation ability of VLP to a certain extent. On the other hand, compared with the training dataset used by the VLP model (taking the CLIP model as an example, its text-image data scale is 400 million), the downstream cross-modal retrieval dataset is usually very small (taking the text-image person re-identification dataset CUHK-PEDES as an example, its scale is only about 80,000), and in the downstream dataset, which is not large in scale, there are many problems such as inaccurate text description and single word usage caused by human factors. Training on such a small-scale downstream dataset with a lot of noise can easily lead to overfitting of the VLP model. Summary of the invention

[0004] The purpose of the present invention is to provide a method for constructing a cross-modal representation model based on text attention and a retrieval method, so as to solve the problems that the VLP model loses its powerful representation ability on smaller downstream data sets and the text description in the data set is single and erroneous, so as to improve the retrieval accuracy.

[0005] To achieve the above object, the technical solution adopted by the present invention is: A method for constructing a cross-modal representation model based on text attention, wherein the cross-modal representation model framework adopted by the method consists of an online model and a momentum model with the same model architecture, wherein the momentum model is a smooth and stable version of the online model; the architectures of the online model and the momentum model both include an image encoder, a text encoder, and a cross-modal encoder; the online model uses an image-text contrast learning module, an image-text matching learning module, and a mask language modeling module, and adds an attention-guided mask modeling module and a text enhancement module for training; Specifically, the method updates the momentum model through the exponential moving average of the online model parameters, and then uses the features encoded by the momentum model as pseudo labels to guide the online model to better characterize learning; wherein the learning and training process steps of the online model are as follows: Step 1: For the image-text pairs in a given dataset , the image and text Input into the image encoder and text encoder respectively to obtain image encoding features , text encoding features and the attention weights during text encoding, where and are the number of image patches and text words, respectively. and As a global feature representation; Step 2: Image-text contrast learning module based on image encoding features and text encoding features Calculate the contrastive learning loss of image text, i.e. ITC loss; Step 3: Input the text and the attention weights obtained in the text encoding process in step 1 into the attention-guided mask modeling module. The attention-guided mask modeling module selects the words in the text T according to the attention weights and replaces them with the [mask] mark to obtain the text with the [mask] mark replaced; input the text with the [mask] mark replaced into the text encoder to obtain the text encoding features with the [mask] mark replaced; Step 4: The text encoding features and image encoding features obtained in step 1 are fed into the cross-modal encoder, with the text encoding features as the query vector, the image encoding features as the key vector and the value vector, and the encoding vector of the last layer of the cross-modal encoder as the output feature of the cross-modal encoder; the output feature of the cross-modal encoder is fed into the image-text matching learning module to calculate the ITM loss; Step 5: Input the image encoding features obtained in step 1 and the text encoding features replaced with the [mask] tag obtained in step 3 into the cross-modal encoder, use the text encoding features replaced with the [mask] tag as the query vector, use the image encoding features as the key vector and the value vector, and use the encoding vector of the last layer of the cross-modal encoder as the output of the cross-modal encoder; input the output features of the cross-modal encoder into the attention-guided mask modeling module, and the attention-guided mask modeling module sends the text encoding features replaced with the [mask] tag to a fully connected layer, and the output is the probability logit of the word at that position belonging to any word in the vocabulary, and calculate the MLM loss based on the logit and the original word before being replaced with the [mask] tag; Step 6: Input the logit value obtained in step 5 into the text enhancement module. The text enhancement module replaces the words marked with [mask] in the original description with words with higher logit values ​​at a certain probability according to the logit value. After performing the above replacement operation on all the words marked with [mask] in the text, a new text is obtained, and then the new text is used as a positive sample of the corresponding image. Step 7: Use the sum of ITC, ITM, and MLM losses as the total loss, and use the result of momentum model encoding as a pseudo-label to guide the online model update; specifically, the total loss gradient is back-propagated to update the text encoder, image encoder, and cross-modal encoder of the online model, and the momentum model is updated according to the online model without the need for gradient back-propagation; Step 8. Iterate the image-text pairs in the dataset and repeat steps 1-7 to perform learning and training of the online model until the preset number of training times or the training end condition is reached. After the training is completed, the image encoder and text encoder in the online model together constitute a cross-modal representation model.

[0006] In step 2, the ITC loss is calculated as follows:

[0007] in, and represents the real one-hot label, and H() represents the entropy function; is the similarity from image to text, is the text-to-image similarity, which is calculated as follows:

[0008] in, represents the temperature parameter, Represents the similarity function.

[0009] In step 3, the attention-guided mask modeling module selects words in the text T according to the attention weights and replaces them with the [mask] flag as follows: For text , the attention weight of the [CLS] tag at each layer is expressed as:

[0010] Among them, A CLS represents the set of attention weights corresponding to the [CLS] tokens of all layers of the text encoder, refers to the attention weight corresponding to the [CLS] tag in layer i, is the number of layers in the text encoder; The attention-guided mask modeling module uses an exponential moving average to combine the attention weights of different layers of the text encoder:

[0011] in, is the set value, , Table 2 refers to the attention weights of the [CLS] tag in the k-th layer text encoder. , , , , , is through The intermediate results are obtained by weighting. The final [CLS] tag corresponds to the attention weight; The attention weights are re-performed with a softmax operation, expressed as follows:

[0012] in, The corresponding attention weight of word i, It is a temperature parameter used to dynamically adjust the range and interval of softmax results. is the total number of inputs; The value after the softmax operation is used as a reference to mask the corresponding words. Specifically, the probability of masking the i-th word is expressed as:

[0013] in, is the offset, is the amplitude; Finally, the word is selected based on the obtained probability and replaced with a special [mask] tag.

[0014] In step 4, the ITM loss is calculated as follows:

[0015] Among them, y itm represents the true label, which is 1 if the text image corresponds and 0 if the text image does not correspond; p itm It indicates the probability of the model outputting whether the text and the image correspond. The ultimate optimization goal is to make the model output close to 1 for positive sample pairs and close to 0 for negative sample pairs. H() represents the entropy function. In step 5, the MLM loss is calculated as follows:

[0016] Among them, y msk represents the true label of the original word corresponding to the word replaced by the [mask] tag, p msk Represents the probability of the model output corresponding to the real word, Indicates the text with the [mask] tag replaced.

[0017] An instance retrieval method based on a cross-modal representation model, the retrieval method comprising the following steps: Step 1: First, input all the pictures in the picture library into the image encoder of the cross-modal representation model for encoding to obtain picture representations, and store all picture representations in the vector database; Step 2: Input the natural language description of the target image of interest into the text encoder of the cross-modal representation model as a query to obtain a text representation; Step 3: Calculate the distance between the text representation obtained in step 2 and the image representation in the vector database, and output the N images that are most similar to the text description; The cross-modal representation model is obtained by using the cross-modal representation model construction method based on text attention as described above.

[0018] After adopting the above scheme, the present invention adds two highly innovative modules to the online model: an attention-guided mask modeling module and a text enhancement module. Among them, the attention-guided mask modeling module uses the attention in the forward process of text encoding as a reference to adaptively mask the words in the text description. In this way, words with practical meanings have a higher probability of being selected for masking than words without practical meanings. In this way, the mask language modeling module can capture the semantics associated with the masked words for alignment during the process of predicting the masked words, thereby more efficiently aligning the cross-modal representations and preventing the loss of VLP's powerful semantic understanding ability. The text enhancement module expands the text description in the downstream task dataset in a lightweight way. Specifically, in the process of predicting the original masked word, the masked language modeling module will generate a probability value for the word belonging to each word in the vocabulary. Thanks to the powerful semantic understanding ability of VLP and the information of the image seen in the prediction process, words with high probability values ​​can also describe the content in the original image well, and even better conform to the corresponding semantics in the image. Therefore, the text enhancement module replaces the masked word with the word corresponding to the high probability value as the new text description of the image, thereby solving the problems of single wording and errors in the text description in the dataset and improving the retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a principle block diagram of the present invention; Figure 2 It is a comparison chart between the ordinary MLM method and the AGM method; Figure 3 Visualize the result graph for cross-modal attention; Figure 4 This is a schematic diagram of the text enhancement module processing; Figure 5 Retrieve the flowchart for the instance. DETAILED DESCRIPTION

[0020] like Figure 1As shown, the present invention discloses a method for constructing a cross-modal representation model based on text attention. The cross-modal representation model framework adopted by the method consists of an online model and a momentum model with the same model architecture. The momentum model is a smooth and stable version of the online model. The architectures of the online model and the momentum model both include an image encoder, a text encoder, and a cross-modal encoder, but the learning and optimization processes of the two are different. The momentum model adopts a basic model (including an image-text contrast learning module (ITC), an image-text matching learning module (ITM), and a mask language modeling module (MLM)), and is updated by the exponential moving average (EMA) of the online model parameters. The online model uses an image-text contrast learning module (ITC), an image-text matching learning module (ITM), and a mask language modeling module (MLM), and adds an attention-guided mask modeling module (AGM) and a text enhancement module (TEM) for training.

[0021] Based on the above cross-modal representation model framework, the cross-modal representation model construction method of the present invention is as follows: the momentum model is updated by the exponential moving average (EMA) of the online model parameters, and then the features encoded by the momentum model are used as pseudo labels to guide the online model to better represent learning. Among them, the learning and training process of the momentum model is the same as the existing method, and the present invention will not elaborate on this. The learning and training process steps of the online model are as follows: Step 1: Given an image-text pair , the image and text Input into the image encoder and text encoder respectively to obtain image encoding features , text encoding features and the attention weights during text encoding, where and are the number of image patches and text words, respectively. and As a global feature representation.

[0022] Step 2: Image-text contrast learning module (ITC) uses image encoding features and text encoding features Compute the contrastive learning loss for image-text.

[0023] Specifically, the softmax-normalized text-to-image similarity and image-to-text similarity are calculated as follows:

[0024] in, is the similarity from image to text, is the text-to-image similarity, represents the temperature parameter, Represents the similarity function, M represents the size of a Mini-batch in a training process. Model training in deep learning divides the data set into batches for training.

[0025] The overall ITC loss is expressed as follows:

[0026] in, and represents the true one-hot label, and H( ) represents the entropy function.

[0027] Step 3: The attention weights in the text encoding process obtained in step 1 are input into the attention-guided mask modeling module (AGM). The attention-guided mask modeling module selects words in the text T according to the attention weights and replaces them with the [mask] mark to obtain the text with the [mask] mark replaced; the text with the [mask] mark replaced is input into the text encoder to obtain the text encoding features with the [mask] mark replaced.

[0028] (1) Limitations of ordinary MLM. Masked language modeling (MLM) aims to predict a masked subset of the input token based on the text context. Ordinary MLM adopts a random masking strategy, which is usually set to have a 15% probability of each word being selected. Among the selected words, 80% are replaced with the [MASK] tag, 10% are randomly replaced with other words, and 10% remain unchanged. MLM works in a cross-modal interaction encoder. The cross-modal interaction encoder consists of a multi-head self-attention layer (MHSA), a multi-head cross-attention (MHCA) layer, and a feed-forward network (FFN). Since MHSA and FFN do not contain the interaction between the two modalities, only MHCA is considered here. Assume Indicates masked text. The processing of MHCA is as follows:

[0029] in, Refers to the The query matrix generated by the masked text in the cross-modal encoder, and Refers to the The key and value matrix generated by the image token in the cross-modal encoder of the last layer. After that, the features corresponding to the [MASK] token in the encoded features of the last layer will be sent to the MLM classification head to predict the original words. These masked tokens are used as the benchmark, and the semantic information related to these masked tokens is captured from the text context through the attention mechanism to achieve the purpose of cross-modal representation alignment. For example, Figure 3As shown in , when the masked word is "heels", the "white chunk" in the text and the related semantics in the image must be captured to predict the original word "heels". However, ordinary cross-modal MLM methods all adopt a random masking strategy, which results in a large part of the masked words being words without practical meaning, such as "the", "also", "with", etc. These words have no relevant semantics in the text context and the image, so it is impossible to align the cross-modal feature representation. The cross-attention visualization diagram between the masked words and the image blocks can reflect this problem. As shown in Figure 3 As shown, when the masked word is “with”, no valuable visual semantic information is captured to help align cross-modal feature representations.

[0030] (2) Attention-guided mask modeling. Figure 2 As shown in the figure, the attention-guided mask modeling module (AGM) is designed to better align cross-modal representations. The ordinary MLM method uses the same mask probability for all words, while the attention-guided mask modeling module uses dynamic probability to adaptively mask words. As we all know, in the process of text encoding, the feature corresponding to the [CLS] tag will be used as a global feature representation, which is the result of weighting all words. Therefore, the corresponding attention weight can reflect the importance of each word, and indirectly reflect whether the word has rich information, so that it can be used as a reference for the probability of the word being masked. Therefore, the attention-guided mask modeling module uses the attention weight corresponding to the [CLS] tag of the text as a reference to dynamically adjust the probability of a word being masked. Specifically, for the text , the attention weight of the [CLS] tag at each layer can be expressed as:

[0031] Among them, A CLS represents the set of attention weights corresponding to the [CLS] tokens of all layers of the text encoder, refers to the attention weight corresponding to the [CLS] tag in layer i, is the number of layers in the text encoder.

[0032] The most intuitive strategy is to average each layer to obtain the final attention weights, but as the network becomes deeper, the noise will decrease and the network can pay more attention to semantically rich words, that is, the attention weights of deep layers are more valuable than those of shallow layers. Therefore, the attention-guided mask modeling module uses an exponential moving average to combine the attention weights of different layers of the text encoder:

[0033] The final design is set , Table 2 refers to the attention weights of the [CLS] tag in the k-th layer text encoder. , , , , , is through The intermediate results are obtained by weighting. is the final [CLS] tag corresponding to the attention weight. It is worth noting that The self-attention weight of the [CLS] tag is included, but the [CLS] tag does not have an actual word corresponding to it, so the attention weight needs to be re-softmaxed to ignore the influence of the class token's self-attention. This operation can be expressed as follows:

[0034] in, The corresponding attention weight of word i, It is a temperature parameter used to dynamically adjust the range and interval of the softmax result. In this example, it is set to 0.02. is the total number of inputs (number of words + one [CLS] tag). The value after softmax is used as a reference to mask the corresponding words. Specifically, the probability of masking the i-th word is expressed as:

[0035] in, is the offset, is the amplitude, both of which can be set to control the proportion of words that are blocked in the entire text.

[0036] Finally, the word is selected and replaced with a special [mask] tag according to the obtained probability. Specifically, for each word, a random number between 0 and 1 is randomly generated. If the random number is less than the obtained probability value, the word is replaced with a special [mask] tag.

[0037] Step 4: The text encoding features and image encoding features obtained in step 1 are fed into the cross-modal encoder. The text encoding features are used as query vectors, and the image encoding features are used as key vectors and value vectors. The encoding vector of the last layer of the cross-modal encoder (i.e., the output of the cross-modal encoder) is expressed as ,in, It is the encoding feature corresponding to the [CLS] tag after it passes through the cross-modal encoder. Indicates the corresponding encoding features of each word after passing through the cross-modal encoder.

[0038] The features output by the cross-modal encoder are sent to the ITM module to calculate the ITM loss and perform gradient backpropagation training on the model. The ITM loss calculation formula is shown in the following formula (3).

[0039]

[0040] Among them, ITM loss is the text-image pair matching loss, y itm represents the true label (1 if the text image corresponds, 0 if the text image does not correspond), p itm It indicates the probability of the model outputting whether the text and the image correspond. The ultimate optimization goal is to make the model output close to 1 for positive sample pairs and close to 0 for negative sample pairs.

[0041] Step 5: Input the image encoding features obtained in step 1 and the text encoding features with the [mask] tag replaced obtained in step 3 into the cross-modal encoder. The text encoding features with the [mask] tag replaced are used as the query vector, and the image encoding features are used as the key vector and value vector. The encoding vector of the last layer of the cross-modal encoder (i.e., the output of the cross-modal encoder) is expressed as ,in, It is the encoding feature corresponding to the [CLS] tag after it passes through the cross-modal encoder. Represents the corresponding encoding features of each word (part of which is replaced by the [mask] tag) after passing through the cross-modal encoder.

[0042] The features output by the cross-modal encoder are input to the MLM module to calculate the MLM loss. Specifically, the MLM module will send the encoded features replaced with the [mask] tag to a fully connected layer, and the output is the probability (logit) that the word at that position belongs to any word in the vocabulary, and the MLM loss is calculated based on the logit and the standard answer (the original word before being replaced with the [mask] tag). The MLM loss is shown in the following formula (4).

[0043]

[0044] Among them, y msk represents the true label of the original word corresponding to the word replaced by the [mask] tag, p msk Represents the probability of the model output corresponding to the real word, Represents the text with [mask] tags replaced (this is the input to the model).

[0045] Step 6. Input the probability (logit) of the word replaced with the [mask] mark obtained in step 5 belonging to any word in the vocabulary into the text enhancement module (TEM). The text enhancement module replaces the words replaced with the [mask] mark in the original description with words with higher logit values ​​with a certain probability according to the logit value (the logit value reflects the similarity with the original replaced word. The higher the logit value, the closer the meaning is to the original word). After performing the above replacement operation on all the words replaced with the [mask] mark in the text, a new text is obtained. This new text will be used as a positive sample of the corresponding image (which means it can also appropriately describe the image) to train the model.

[0046] Specifically, inaccurate descriptions are unavoidable in text descriptions. Figure 4 As shown in the figure, the jeans described as "black" in the original description are actually more appropriately described as "blue". On the other hand, the descriptions of the same object are relatively simple. For example, "shoes" is more commonly used, while "sneakers" is rarely seen in descriptions. Due to the limitations of data size and the single and incorrect words used in text descriptions caused by various human factors, VLP is prone to overfitting when trained on the downstream TBPS dataset, thereby losing VLP's powerful semantic understanding ability. To solve this problem, based on the fact that AGM can select more informative words for masking, we designed a text expansion module (TEM) to provide more diverse and high-quality text descriptions. Since VLP has strong semantic understanding capabilities, the logit value of the MLM classification head can reflect the semantic similarity between the predicted word and the original masked word, and thanks to the fact that MLM uses image information when predicting the original word, these words with higher logits can even more accurately describe the corresponding content in the image. As shown in the figure, Figure 4 As shown in the figure, the top-5 predicted word list and the original masked words are shown. It is obvious that these words seem to be suitable for describing the content in the image, or even better. For example, "dark" may be more suitable for describing the color of pants than the original "black". The probability of replacing the masked word with a word with a higher logit is to enrich the original text description and reduce noise without consuming any additional resources. and its corresponding predicted logit value , you can construct a sorted list:

[0047] in, refers to the softmax function, represents the top-m truncation function, which can select the m words with the highest logit. In this way, the masked text is obtained. A list of potential replacements is used to randomly sample a word from it, and the sampled word is guaranteed to be different from the original word. This process can be formalized as:

[0048] Among them, W i Represents the specific word to be replaced, Multinomial represents the multinomial distribution sampling function, and the normalized logit value obtained according to the previous formula is used as its distribution function. The above formula means that the replaced word needs to be sampled according to the multinomial distribution.

[0049] Then, the original words are replaced with new words to obtain new text descriptions. Replacing meaningful words with synonyms not only retains the original information and makes the text description richer, but also makes it more accurate for the corresponding image. This is because in the process of predicting the original words, not only the text context information but also the image information is used. The new description obtained will be used as a positive sample of the corresponding image for representation learning.

[0050] Step 7. Update of online model In addition to the ITC, ITM, and MLM losses mentioned in the above steps, the result of the momentum model encoding will be used as a pseudo-label to guide the update of the online model. This is consistent with the basic model (ALBEF) used in the patent and will not be repeated here. Therefore, the total loss of the final training model is:

[0051] The total loss gradient backpropagation updates the text encoder, image encoder, and cross-modal encoder. The momentum model is updated according to the online model and does not need to be updated through gradient backpropagation.

[0052] Step 8: Iterate the image-text pairs in the dataset and repeat steps 1-7 to perform learning and training of the online model until the preset number of training times or the training end condition is reached. After the training is completed, the image encoder and text encoder in the online model together constitute a cross-modal representation model.

[0053] After the cross-modal representation model is constructed using the above method, the cross-modal representation model is directly applied to the instance retrieval system. The input of the instance retrieval system is the text description, and the output is the N images in the image library that are most similar to the text description. Figure 5 As shown: Step 1: First, input all the pictures in the picture library into the image encoder of the cross-modal representation model for encoding, obtain the picture representation, and store all the picture representations into the vector database.

[0054] Step 2: Input the natural language description of the target image of interest into the text encoder of the cross-modal representation model as a query to obtain a text representation.

[0055] Step 3: Calculate the distance between the text representation obtained in step 2 and the image representation in the vector database, and output the N images that are most similar to the text description.

[0056] The method of the present invention has achieved the best performance so far on three mainstream text image person re-identification datasets (CUHK-PEDES, ICFG-PEDES, and RSTPReid).

[0057] The method of the present invention comprises two highly innovative modules: an attention-guided mask modeling module and a text enhancement module.

[0058] The attention-guided mask modeling module uses the attention in the forward process of text encoding as a reference to adaptively mask the words in the text description. In this way, words with practical meanings are more likely to be selected for masking than words without practical meanings (because meaningful words have higher attention weights). In this way, the masked language modeling module can capture the semantics associated with the masked words for alignment during the prediction of masked words, thereby more efficiently aligning cross-modal representations and preventing the loss of VLP's powerful semantic understanding ability. Taking the results on the CUHK-PEDES dataset as an example, after adding this module, the R@1 accuracy increased by 1.07%.

[0059] The text enhancement module expands the text description in the downstream task dataset in a lightweight way. Specifically, when predicting the original masked word, the masked language modeling module generates the probability value of the word belonging to each word in the vocabulary. Thanks to the powerful semantic understanding ability of VLP and the information of the image seen in the prediction process, the words with high probability values ​​can also well describe the content in the original image, and even better conform to the corresponding semantics in the image. Therefore, the text enhancement module replaces the masked words with the words corresponding to the high probability values ​​as the new text description of the image, thus solving the problems of single words and errors in the text description in the dataset. After adding the text enhancement module, the R@1 accuracy on the CUHK-PEDES dataset increased by 0.36%. When the two modules work together, the R@1 accuracy on the CUHK-PEDES dataset increased by 1.85%.

[0060] The present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments. It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied. Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure. Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for constructing a cross-modal representation model based on text attention, characterized by: The cross-modal representation model framework adopted by the method is composed of an online model and a momentum model with the same model architecture, and the momentum model is a smooth and stable version of the online model; the architectures of the online model and the momentum model both include an image encoder, a text encoder, and a cross-modal encoder; the online model uses an image-text contrast learning module, an image-text matching learning module, a mask language modeling module, and adds an attention-guided mask modeling module and a text enhancement module for training; Specifically, the method updates the momentum model through the exponential moving average of the online model parameters, and then uses the features encoded by the momentum model as pseudo labels to guide the online model to better characterize learning; wherein the learning and training process steps of the online model are as follows: Step 1: For the image-text pairs in a given dataset , the image and text Input into the image encoder and text encoder respectively to obtain image encoding features , text encoding features and the attention weights during text encoding, where and are the number of image patches and text words, respectively. and As a global feature representation; Step 2: Image-text contrast learning module based on image encoding features and text encoding features Calculate the contrastive learning loss of image text, i.e. ITC loss; Step 3: Input the text and the attention weights obtained in the text encoding process in step 1 into the attention-guided mask modeling module. The attention-guided mask modeling module selects the words in the text T according to the attention weights and replaces them with the [mask] mark to obtain the text with the [mask] mark replaced; input the text with the [mask] mark replaced into the text encoder to obtain the text encoding features with the [mask] mark replaced; Step 4: The text encoding features and image encoding features obtained in step 1 are fed into the cross-modal encoder, with the text encoding features as the query vector, the image encoding features as the key vector and the value vector, and the encoding vector of the last layer of the cross-modal encoder as the output feature of the cross-modal encoder; the output feature of the cross-modal encoder is fed into the image-text matching learning module to calculate the ITM loss; Step 5: Input the image encoding features obtained in step 1 and the text encoding features replaced with the [mask] tag obtained in step 3 into the cross-modal encoder, use the text encoding features replaced with the [mask] tag as the query vector, use the image encoding features as the key vector and the value vector, and use the encoding vector of the last layer of the cross-modal encoder as the output of the cross-modal encoder; input the output features of the cross-modal encoder into the attention-guided mask modeling module, and the attention-guided mask modeling module sends the text encoding features replaced with the [mask] tag into a fully connected layer, and the output is the probability logit of the word at that position belonging to any word in the vocabulary, and calculate the MLM loss based on the logit and the original word before being replaced with the [mask] tag; Step 6: Input the logit value obtained in step 5 into the text enhancement module. The text enhancement module replaces the words marked with [mask] in the original description with words with higher logit values ​​at a certain probability according to the logit value. After performing the above replacement operation on all the words marked with [mask] in the text, a new text is obtained, and then the new text is used as a positive sample of the corresponding image. Step 7: Use the sum of ITC, ITM, and MLM losses as the total loss, and use the result of momentum model encoding as a pseudo-label to guide the online model update; specifically, the total loss gradient is back-propagated to update the text encoder, image encoder, and cross-modal encoder of the online model, and the momentum model is updated according to the online model without the need for gradient back-propagation; Step 8. Iterate the image-text pairs in the dataset and repeat steps 1-7 to perform learning and training of the online model until the preset number of training times or the training end condition is reached. After the training is completed, the image encoder and text encoder in the online model together constitute a cross-modal representation model.

2. The method for constructing a cross-modal representation model based on text attention according to claim 1, characterized in that: In step 2, the ITC loss is calculated as follows: in, and represents the real one-hot label, and H() represents the entropy function; is the similarity from image to text, is the text-to-image similarity, which is calculated as follows: in, represents the temperature parameter, Represents the similarity function.

3. The method for constructing a cross-modal representation model based on text attention according to claim 1, characterized in that: In step 3, the attention-guided mask modeling module selects words in the text T according to the attention weights and replaces them with the [mask] flag as follows: For text , the attention weight of the [CLS] tag at each layer is expressed as: Among them, A CLS represents the set of attention weights corresponding to the [CLS] tokens of all layers of the text encoder, refers to the attention weight corresponding to the [CLS] tag in layer i, is the number of layers in the text encoder; The attention-guided mask modeling module uses an exponential moving average to combine the attention weights of different layers of the text encoder: in, is the set value, , Table 2 refers to the attention weights of the [CLS] tag in the k-th layer text encoder. , , , , , is through The intermediate results are obtained by weighting. is the final [CLS] tag corresponding to the attention weight; The attention weights are re-performed with a softmax operation, expressed as follows: in, The corresponding attention weight of word i, It is a temperature parameter used to dynamically adjust the range and interval of softmax results. is the total number of inputs; The value after the softmax operation is used as a reference to mask the corresponding words. Specifically, the probability of masking the i-th word is expressed as: in, is the offset, is the amplitude; Finally, the word is selected based on the obtained probability and replaced with a special [mask] tag.

4. The method for constructing a cross-modal representation model based on text attention according to claim 1, characterized in that: In step 4, the ITM loss is calculated as follows: Among them, y itm represents the true label, which is 1 if the text image corresponds and 0 if the text image does not correspond; p itm It indicates the probability of the model outputting whether the text and the image correspond. The ultimate optimization goal is to make the model output close to 1 for positive sample pairs and close to 0 for negative sample pairs. H() represents the entropy function. In step 5, the MLM loss is calculated as follows: Among them, y msk represents the true label of the original word corresponding to the word replaced by the [mask] tag, p msk Represents the probability of the model output corresponding to the real word, Indicates the text with the [mask] tag replaced.

5. An instance retrieval method based on a cross-modal representation model, characterized in that: The retrieval method comprises the following steps: Step 1: First, input all the pictures in the picture library into the image encoder of the cross-modal representation model for encoding to obtain picture representations, and store all picture representations in the vector database; Step 2: Input the natural language description of the target image of interest into the text encoder of the cross-modal representation model as a query to obtain a text representation; Step 3: Calculate the distance between the text representation obtained in step 2 and the image representation in the vector database, and output the N images that are most similar to the text description; The cross-modal representation model is obtained by using a text attention-based cross-modal representation model construction method as described in any one of claims 1-4.

Images