An image-text matching method based on false negative elimination strategy

Abstract

The present invention discloses an image-text matching method based on a false negative elimination strategy. During the training process, when the number of negative samples is small, the number of false negatives will also be smaller. Therefore, in the training phase of the image encoding module and the text encoding module, in order to obtain more negative samples without increasing the burden on the GPU memory, a momentum memory module is introduced. The module consists of an image encoder with a momentum update method, namely a momentum image encoder, a momentum image text encoder, and two memory banks, namely an image memory bank M. v 、Text Memory M t At the same time, the false negative elimination strategy uses the similarity between the anchor point and the global features of the negative sample to calculate the probability that the negative sample is a false negative. This weight is then calculated based on the false negative probability. Finally, the negative sample to be pushed aside is selected based on the weight for use in the loss function calculation. The higher the false negative probability, the lower the sampling weight, thereby achieving false negative elimination. This way, when selecting negative samples to be pushed aside, the false negative probability is calculated to minimize their selection, achieving better model training and obtaining more discriminative modal features.

Description

Technical Field

[0001] The present invention belongs to the technical field of image-text matching, and more specifically, relates to an image-text matching method based on a false negative elimination strategy. Background Art

[0002] Image-text matching is a fundamental yet critical and challenging problem in the field of vision and language. Its goal is to retrieve exact matches for a given query, i.e., searching for text with images and searching for images with text.

[0003] Existing image-text matching methods can be divided into intra-modal interaction-based and inter-modal interaction-based methods based on different modal interactions. The intra-modal interaction-based method uses independent image and text encoders to understand and explore semantic clues within each modality. However, such an intra-modal interaction framework lacks information interaction between modalities, resulting in less than ideal final performance. Therefore, an inter-modal-based method was proposed, which usually uses a deep network with a large number of inter-modal interaction operations, takes images and text as input, and comprehensively explores the interaction between them to achieve better image-text matching. But precisely because of this, a large number of interactive operations greatly increases the computational cost of the reasoning process, ultimately leading to high retrieval latency.

[0004] Whether based on intra-modal or inter-modal interaction, the model training process uses a triplet loss function to optimize the model. A triplet consists of an anchor point, a negative sample, and a positive sample. The model performs specific training optimization by shortening the distance between the anchor point and the positive sample and increasing the distance between the anchor point and the negative sample. The formula is as follows:

[0005]

[0006] Among them, margin is a constraint hyperparameter, s() is the cosine similarity, and The image features and text features of the anchor samples, and Indicates the negative sample image features and text features that are closest to the anchor sample image features and text features in the current batch. It can be seen that the negative sample selected to be pushed is the negative sample that is closest to the anchor sample features. However, due to the diversity of semantics, and in the corresponding dataset, only the samples that match when the initial dataset is constructed are considered positive samples, and all other samples are considered negative samples. Therefore, among these negative samples, there may still be samples that match the anchor in semantic content, that is, the existence of false negatives, such as Figure 1 The middle black arrow in the figure shows the part. False negative samples are samples that are related to the anchor point but are incorrectly labeled as negative. If these false negative samples are excluded, it will create conflicting optimization objectives and hinder model training. Summary of the Invention

[0007] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an image-text matching method based on a false negative elimination strategy. During the training process, when selecting negative samples to be pushed out, a false negative elimination strategy is proposed by calculating the false negative probability to avoid the selection of false negatives as much as possible, aiming to reduce the presence of false negative samples in the loss function, achieve better model training, and obtain more discriminative modal features.

[0008] To achieve the above-mentioned object of the invention, the image-text matching method based on the false negative elimination strategy of the present invention is characterized by comprising the following steps:

[0009] (1) Training phase

[0010] 1.1) Extracting image text features

[0011] Select a batch of b images from the training dataset {I 1 , I 2 ,…,I x ,…,I b} and b images, each of which corresponds to a text describing the image {T 1 , T 2 ,…,T x ,…,T b};

[0012] 1.1.1) Extraction of image features

[0013] The image encoder is used to extract the image features of b images. For the xth image, the image is divided into multiple image blocks for feature embedding, thereby forming the image feature V x :

[0014]

[0015] in, is the i-th image block feature of the x-th image Ix, m is the number of image blocks, and the average pooling operation is used to obtain the global feature of the x-th image Ix

[0016]

[0017] 1.1.2) Extraction of text features

[0018] For b texts, a text encoder is used to extract text features. For the xth text T x Extract the feature representation of each word in the text to form the text feature W x :

[0019]

[0020] in, is the jth word feature of the xth text Tx, l is the number of words, and the xth text T is obtained by average pooling operation x Global characteristics of

[0021]

[0022] 1.1.3) Extraction of momentum features

[0023] At the same time, use the momentum image encoder to obtain the momentum image features according to the method of step 1.1.1) for b images And store it in the image memory M v middle;

[0024] Use the momentum text encoder to get the momentum text features according to the method of step 1.1.2) for b texts And store it in text memory M t ;

[0025] Image Memory M v With text memory M t Each is a queue, and the queue length is K;

[0026] 1.2) Calculate image-text similarity

[0027] For the xth image I x Global characteristics of Calculate its relationship with the text memory M t The similarity of each momentum text feature in For the xth text T x Global characteristics of Calculate its relationship with the image memory M v The similarity of each momentum image feature in

[0028] 1.3) Eliminate false negative samples

[0029] 1.3.1) Calculate the similarity distribution of matching pairs and non-matching pairs

[0030] The similarity Based on whether the image and text are matching pairs or non-matching pairs, the similarity of matching pairs and non-matching pairs are divided. Combined with the similarity of matching pairs and non-matching pairs accumulated in the previous batch, the mean similarity μ of matching pairs is calculated. + and variance σ + and the mean similarity ν of the unmatched pairs -and variance σ - ;

[0031] Construct matching pair similarity distribution f S|c (s):

[0032]

[0033] Constructing the similarity distribution of mismatched pairs

[0034]

[0035] Among them, c represents the matching between image and text, Indicates that the image and text do not match, S represents a random variable about the similarity between the image and the text, and s represents the similarity;

[0036] 1.3.2) Calculate the posterior probability P(C=c|S=s):

[0037]

[0038] Where P(c)=p, p is the set hyperparameter;

[0039] 1.3.3) Calculate the sampling weight of the xth anchor point

[0040] According to the posterior probability P(C=c|S=s), for the x-th anchor point, it is recorded as a x , the global feature of the i-th negative sample of the other modality The sampling weight for:

[0041]

[0042] Among them, λ is set to 0.01, d + is the distance from anchor point a x The global features of the most distant positive sample of the other modality;

[0043] The anchor point is the xth image I x Global Features When the i-th negative sample of the other modality For the xth image I x The global features of the i-th negative sample text, similarly, the anchor point is the x-th text T x Global Features When the i-th negative sample of the other modality is the global feature of the i-th negative sample image of the x-th text Tx, s(*), *) represents the similarity of the global feature;

[0044] 1.3.4) Sampling to obtain global features of negative sample images and text

[0045] According to the sampling weight, the global features of the negative sample text or image are sampled to obtain the xth image I x The global features of all negative sample texts are obtained by sampling according to the sampling weight The xth text T x The global features of all negative sample images are obtained by sampling according to the sampling weights

[0046] 1.4) Calculate the loss function

[0047]

[0048] Among them, margin is a constraint hyperparameter;

[0049] 1.5) Training image encoder, text encoder, momentum image encoder, momentum text encoder

[0050] For the parameters θ of the image encoder q v , parameters θ of the text encoder q t , by using the loss function Use the back propagation algorithm to calculate the gradient and update the parameters;

[0051] For the momentum image encoder θ k v , parameters θ of the momentum text encoder k t , its update method is no longer based on gradient backpropagation, but is updated through momentum update:

[0052] θ k v ←mθ k v +(1-m)θ q v

[0053] θ k t ←mθ k t +(1-m)θ q t

[0054] m is the momentum update parameter;

[0055] Repeat steps 1.1 to 1.5 until convergence, completing the training of the image encoder and text encoder.

[0056] (2) Reasoning stage

[0057] For image retrieval of text, an image encoder is used to extract the global features of the image, and the similarity is calculated and compared with the global features of all texts in the text library. The matching text with the greatest similarity is selected to complete the matching;

[0058] For text retrieval images, a text encoder is used to extract global features of the text, and the similarity is calculated and compared with the global features of all images in the image library. The matching text with the greatest similarity is selected to complete the matching.

[0059] The object of the present invention is achieved like this:

[0060] The image-text matching method of the present invention is based on the false negative elimination strategy. During the training process, when the number of negative samples is small, the number of false negatives will also be smaller. Therefore, in order to obtain more negative samples without increasing the burden on the GPU memory during the training phase of the image encoding module and the text encoding module, a momentum memory module is introduced. The module consists of an image encoder with a momentum update method, namely the momentum image encoder, the momentum image text encoder, and two memory banks, namely the image memory bank M. v 、Text Memory M t At the same time, the false negative elimination strategy uses the similarity between the anchor point and the global features of the negative sample to calculate the probability that the negative sample is a false negative. This weight is then calculated based on the false negative probability. Finally, the negative sample to be pushed aside is selected based on the weight for use in the loss function calculation. The higher the false negative probability, the lower the sampling weight, thereby achieving false negative elimination. This way, when selecting negative samples to be pushed aside, the false negative probability is calculated to minimize their selection, achieving better model training and obtaining more discriminative modal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the composition of triples for image-text matching;

[0062] Figure 2 This is a flowchart of a specific implementation of the image-text matching method based on the false negative elimination strategy of the present invention;

[0063] Figure 3 This is a schematic diagram of a specific embodiment of the image-text matching method based on the false negative elimination strategy of the present invention;

[0064] Figure 4 It is a schematic diagram of the principle of eliminating false negative samples. DETAILED DESCRIPTION

[0065] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0066] Figure 2 、 3 They are respectively a flow chart and a principle schematic diagram of a specific implementation of the image-text matching method based on the false negative elimination strategy of the present invention.

[0067] In this embodiment, if Figure 2 、 3 As shown, the image-text matching method based on the false negative elimination strategy of the present invention includes a training phase and an inference phase, and the specific steps are as follows:

[0068] Step S1: Training phase

[0069] Step S1.1: Extract image text features

[0070] Select a batch of b images from the training dataset {I 1 , I 2 ,…,I x ,…,I b} and b images, each of which corresponds to a text describing the image {T 1 , T 2 ,…,T x ,…,T b}.

[0071] Step SI.1.1: Image feature extraction

[0072] The image encoder is used to extract the image features of b images. For the xth image, the image is divided into multiple image blocks (patch blocks) for feature embedding, thereby forming the image feature V x :

[0073]

[0074] in, For the xth image I x The i-th image block feature, m is the number of image blocks, and the average pooling operation is used to obtain the x-th image I x Global Features

[0075]

[0076] In this embodiment, the pre-trained visual Transformer model Vision-Transformer is used as the image encoder for all images in the current training batch {I 1 , I 2 ,…,I x ,…,I b} to extract image features, expressed as ViT, so that the image feature V x It can be expressed as:

[0077]

[0078] Step SI.1.2: Extraction of text features

[0079] For b texts, a text encoder is used to extract text features. For the xth text T x Extract the feature representation of each word in the text to form the text feature W x :

[0080]

[0081] in, For the xth text T x The jth word feature of , l is the number of words, and the average pooling operation is used to obtain the xth text T x Global characteristics of

[0082]

[0083] In this embodiment, the pre-trained Transformer model BERT is used as the text encoder to encode all the texts in the current training batch {T 1 , T 2 ,…,T x ,…,T b} to extract features and use BERT to represent the text feature W x It can be expressed as:

[0084] Step S1.1.3: Extraction of momentum features

[0085] If only the global features of the current batch obtained above are used to eliminate false negatives, the effect will be greatly reduced. Because it is intuitive that when the number of negative samples is small, the number of false negatives will also be smaller. Therefore, in the training phase of the image encoding module and the text encoding module, in order to obtain more negative samples without increasing the burden on the GPU memory, a momentum memory module is introduced. This module consists of an image encoder with a momentum update method, namely the momentum image encoder, the momentum image text encoder, and two memory banks, namely the image memory bank M. v 、Text Memory M t composition.

[0086] In this embodiment, the image encoder and text encoder with momentum update are still ViT and BERT models, but their update method is no longer based on gradient feedback, but through momentum update.

[0087] θ k v ←mθ k v +(1-m)θ q v

[0088] θ k t ←mθ k t +(1-m)θ q t

[0089] Among them, θ q v and θ q t are the parameters of the image encoder and text encoder for image-text matching, θ k v and θ k t are the parameters of the momentum image encoder and momentum text encoder, m is the momentum update parameter, set to 0.995. During the training process, θ q v and θ q t The parameters are updated by using the back propagation algorithm to calculate the gradient on the subsequent loss function, and θ k v and θ k t The parameters are updated only through the above momentum update formula.

[0090] The essence of the memory bank is a queue, which contains not only the feature representation of the current batch, but also the feature representation of the previous batch. It is precisely because of the existence of the memory bank that the scope of mining negative samples is no longer limited to the mini-batch size, but is related to the momentum memory module size (queue size). At the same time, more negative samples are obtained without increasing the burden on the GPU memory. When the current batch features are stored in the memory bank, due to the fixed size of the queue, the oldest feature representation in the queue will be dequeued. The image encoder and text encoder with momentum update method can ensure the consistency of the features stored in the memory bank. Specifically:

[0091] Use the momentum image encoder on b images according to the method of step S1.1.1 to obtain the momentum image features And store it in the image memory M v middle.

[0092] Use the momentum text encoder to get the momentum text features according to the method of step S1.1.2 for b texts And store it in text memory M t .

[0093] Image Memory M v With text memory M t Each is a queue, and the queue length is K.

[0094] Step S1.2: Calculate image-text similarity

[0095] For the xth image I x Global characteristics of Calculate its relationship with the text memory M t The similarity of each momentum text feature in For the xth text T x Global characteristics of Calculate its relationship with the image memory M v The similarity of each momentum image feature in

[0096] Step S1.3: Eliminate false negative samples

[0097] During the training phase, the global image and text features extracted in step S1 are used to calculate a loss function to train the image and text encoders. Traditional methods use the triplet loss function described in the background. However, the presence of false negatives creates conflicting optimization objectives, hindering the training of the image and text encoders. Therefore, the present invention proposes a false negative elimination strategy to reduce the presence of false negative samples in the loss function.

[0098] The false negative elimination strategy is to calculate the probability that the negative sample is a false negative through the similarity between the global features of the anchor point and the negative sample, then calculate the weight based on the false negative probability, and finally select the negative sample to be pushed out based on the weight for the calculation of the loss function. Among them, the higher the false negative probability, the lower the sampling weight, so as to achieve false negative elimination. Here, the false negative probability is calculated by similarity, that is, under the premise of knowing the similarity of the global features of the anchor point and the negative sample, the probability of the two matching is judged. Expressed in mathematical language, it is to calculate the posterior probability P(C=c|S=s), where the random variable C represents whether an image matches a text. It is a discrete random variable, and c is the match between the image and the text. S is a random variable about the similarity between the image and the text. It is a continuous random variable, and s is the specific value of the similarity. The posterior probability P(C=c|S=s) can be derived by Bayes' theorem as:

[0099]

[0100] It can be seen that the probability P(C=c|S=s) is finally equal to P(c) and and f S|c (s) and Among them, P(c) and are two random events of C, namely image and text matching and mismatching, c represents image and text matching, Indicates that the image and text do not match. Therefore, the variable C is a Bernoulli distribution, which has only one parameter, the matching probability parameter, which is set as the hyperparameter p, so P(c) = p. And f S|c (s) and The similarity distribution of image-text matching and the similarity distribution of mismatching are constructed by accumulating the sampled similarities in each mini-batch. Figure 4 As shown in , in each batch, the similarity scores of the matching pairs in the current batch and the similarity of the unmatched pairs are sampled. Then, the mean and variance of the similarity between the matching pairs and the unmatched pairs accumulated in the previous batch are calculated, that is, the mean similarity μ of the matching pairs + and variance σ + , the mean similarity μ of the unmatched pairs - and variance σ - After obtaining , the two distributions will be constructed into normal distributions according to the laws of nature, such as Figure 4 As shown by the dotted line, the formula is as follows:

[0101]

[0102]

[0103] At this point, P(c)=p, Substituting the similarity between the anchor point and the negative sample into the above two equations completes the calculation of the false negative probability. After calculating the false negative probability of each negative sample through the above correlation distribution calculation, the sampling weight is calculated using the following formula, so that the higher the false negative probability, the smaller the sampling weight.

[0104]

[0105] Among them, a represents the anchor point, that is, the global feature of the image or text, represents the global feature of the i-th negative sample, p i That is, the calculated weight of whether the current negative sample will be selected as a triplet to be pushed out, such as Figure 4 shown.

[0106] However, using this single formula for weight calculation presents a problem. Since the false negative probability is positively correlated with similarity, the lower the similarity, the lower the false negative probability. However, the lower the false negative probability, the higher the sampling weight, as we are trying to avoid sampling false negatives. Therefore, when the global features of a negative sample have little similarity to the anchor point, its sampling probability is calculated to be higher. However, sampling these overly simple negative samples does not provide sufficient information for model training and optimization, as there is no correlation between the two. Therefore, for negative samples with very low similarity, a truncation strategy is employed, using a different calculation method to calculate the sampling weight.

[0107]

[0108] where λ is set to 0.01, d + is the global feature of the farthest positive sample

[0109] The second line indicates that when the false negative probability is very small, that is, when the similarity is very small, another weight calculation method is used. In this method, the smaller the similarity, the smaller the sampling weight will be, such as Figure 4 shown.

[0110] Thus, the specific steps of the false negative elimination strategy are:

[0111] Step S1.3.1: Calculate the similarity distribution of matching pairs and non-matching pairs

[0112] The similarity Based on whether the image and text are matching pairs or non-matching pairs, the similarity of matching pairs and non-matching pairs are divided. Combined with the similarity of matching pairs and non-matching pairs accumulated in the previous batch, the mean similarity μ of matching pairs is calculated. + and variance σ + and the mean similarity μ of the unmatched pairs- and variance σ - .

[0113] Construct matching pair similarity distribution f S|c (s):

[0114]

[0115] Constructing the similarity distribution of mismatched pairs

[0116]

[0117] Among them, c represents the matching between image and text, Indicates that the image and text do not match, S represents a random variable about the similarity between the image and the text, and s represents the similarity.

[0118] Step S1.3.2: Calculate the posterior probability P(C=c|S=s):

[0119]

[0120] Where P(c)=p, p is the set hyperparameter;

[0121] Step S1.3.3: Calculate the sampling weight of the x-th anchor point

[0122] According to the posterior probability P(C=c|S=s), for the x-th anchor point, it is recorded as a x , the i-th negative sample of the other modality The sampling weight for:

[0123]

[0124] Among them, λ is set to 0.01, For, d + is the distance from anchor point a x The global features of the most distant positive samples of the other modality.

[0125] The anchor point is the xth image I x Global Features When the i-th negative sample of the other modality For the xth image I x The global features of the i-th negative sample text, similarly, the anchor point is the x-th text T x Global Features When the i-th negative sample of the other modality is the global feature of the i-th negative sample image of the x-th text Tx, and s(*, *) represents the similarity of the global features.

[0126] Step S1.3.4: Sampling to obtain global features of negative sample images and text

[0127] According to the sampling weight, the global features of the negative sample text or image are sampled to obtain the xth image I x The global features of all negative sample texts are obtained by sampling according to the sampling weight The xth text T x The global features of all negative sample images are obtained by sampling according to the sampling weights

[0128] Step S1.4: Calculate the loss function

[0129]

[0130] Among them, margin is a constraint hyperparameter.

[0131] Step S1.5: Train the image encoder, text encoder, momentum image encoder, and momentum text encoder

[0132] For the parameters θ of the image encoder q v , parameters θ of the text encoder q t , by using the loss function Backpropagation is used to calculate gradients and update parameters. Training image encoders and text encoders using loss functions is a state-of-the-art technique and will not be described in detail here.

[0133] For the momentum image encoder θ k v , parameters θ of the momentum text encoder k t , its update method is no longer based on gradient backpropagation, but is updated through momentum update:

[0134] θ k v ←mθ k v +(1-m)θ q v

[0135] θ k t ←mθ k t +(1-m)θ qt

[0136] m is the momentum update parameter.

[0137] Repeat steps S1.1 to S1.5 until convergence, completing the training of the image encoder and text encoder.

[0138] Step S2: Inference phase

[0139] For image retrieval of text, an image encoder is used to extract the global features of the image, and the similarity is calculated and compared with the global features of all texts in the text library. The matching text with the greatest similarity is selected to complete the matching.

[0140] For text retrieval images, a text encoder is used to extract global features of the text, and the similarity is calculated and compared with the global features of all images in the image library. The matching text with the greatest similarity is selected to complete the matching.

[0141] The reasoning belongs to the prior art and will not be described in detail here.

[0142] In summary, the false negative elimination strategy proposed in this invention can effectively reduce the presence of false negatives in triples, enable the model to be better trained, learn more discriminative feature representations, and further improve the accuracy of image-text matching tasks.

[0143] Examples

[0144] To evaluate the performance of the model, in this example, the present invention is applied to the commonly used datasets MSCOCO and Flickr30K. In both datasets, each image has five description sentences. The present invention uses the recall rate at K (Recall K) as the evaluation metric. The recall rate measures the percentage of true hits in the top K rankings in the list. A higher recall rate indicates better performance. The experimental results are listed in Tables 1 and 2, where Table 1 shows the experimental results on the Flickr30K and MSCOCO (1k) datasets, and Table 2 shows the experimental results on the MSCOCO (5k) dataset.

[0145]

[0146] Table 1

[0147]

[0148] Table 2

[0149] The experimental results in Tables 1 and 2 show that the proposed method outperforms all baseline models in both datasets. These results demonstrate the superiority of the proposed model with a false negative elimination strategy in the image-text matching task.

[0150] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. An image-text matching method based on a false negative elimination strategy, characterized by comprising the following steps: (1) Training phase 1.1) Extracting image text features Select a batch of b images from the training dataset {I 1 ,I 2 ,…,I x ,…,I b } and b images, each of which corresponds to a text describing the image {T 1 ,T 2 ,…,T x ,…,T b }; 1.1.1) Extraction of image features The image encoder is used to extract the image features of b images. For the xth image, the image is divided into multiple image blocks for feature embedding, thereby forming the image feature V x : in, For the xth image I x The i-th image block feature, m is the number of image blocks, and the average pooling operation is used to obtain the x-th image I x Global Features 1.1.2) Extraction of text features For b texts, a text encoder is used to extract text features. For the xth text T x Extract the feature representation of each word in the text to form the text feature W x : in, For the xth text T x The jth word feature of , l is the number of words, and the average pooling operation is used to obtain the xth text T x Global characteristics of 1.1.3) Extraction of momentum features At the same time, use the momentum image encoder to obtain the momentum image features according to the method of step 1.1.1) for b images And store it in the image memory M v middle; Use the momentum text encoder to get the momentum text features according to the method of step 1.1.2) for b texts And store it in text memory M t ; Image Memory M v With text memory M t Each is a queue with a queue length of K; 1.2) Calculate image-text similarity For the xth image I x Global characteristics of Calculate its relationship with the text memory M t The similarity of each momentum text feature in For the xth text T x Global characteristics of Calculate its relationship with the image memory M v The similarity of each momentum image feature in 1.3) Eliminate false negative samples 1.3.1) Calculate the similarity distribution of matching pairs and non-matching pairs The similarity Based on whether the image and text are matching pairs or non-matching pairs, the similarity of matching pairs and non-matching pairs are divided. Combined with the similarity of matching pairs and non-matching pairs accumulated in the previous batch, the mean similarity μ of matching pairs is calculated. + and variance σ + and the mean similarity μ of the unmatched pairs - and variance σ - ; Construct matching pair similarity distribution f S|c (s): Constructing the similarity distribution of mismatched pairs in, c indicates that the image matches the text, Indicates that the image and text do not match, S represents a random variable about the similarity between the image and the text, and s represents the similarity; 1.3.2) Calculate the posterior probability P(C=c|S=s): Where P(c)=p, p is the set hyperparameter; 1.3.3) Calculate the sampling weight of the xth anchor point According to the posterior probability P(C=c|S=s), for the x-th anchor point, it is recorded as a x , the global feature of the i-th negative sample of the other modality The sampling weight for: Among them, λ is set to 0.01, d + is the distance from anchor point a x The global features of the most distant positive sample of the other modality; The anchor point is the xth image I x Global Features When the i-th negative sample of the other modality For the xth image I x The global features of the i-th negative sample text, similarly, the anchor point is the x-th text T x Global Features When the i-th negative sample of the other modality For the xth text T x The global features of the i-th negative sample image, s(*,*) represents the similarity of the global features; 1.3.4) Sampling to obtain global features of negative sample images and text According to the sampling weight, the global features of the negative sample text or image are sampled to obtain the xth image I x The global features of all negative sample texts are obtained by sampling according to the sampling weight The xth text T x The global features of all negative sample images are obtained by sampling according to the sampling weights 1.4) Calculate the loss function Among them, margin is a constraint hyperparameter; 1.5) Training image encoder, text encoder, momentum image encoder, momentum text encoder For the parameters θ of the image encoder q v , parameters θ of the text encoder q t , by using the loss function Use the back propagation algorithm to calculate the gradient and update the parameters; For the momentum image encoder θ k v , parameters θ of the momentum text encoder k t , its update method is no longer based on gradient backpropagation, but is updated through momentum update: i k v ←mθ k v +(1-m)θ q v i k t ←mθ k t +(1-m)θ q t m is the momentum update parameter; Repeat steps 1.1 to 1.5 until convergence, completing the training of the image encoder and text encoder. (2) Reasoning stage For image retrieval of text, an image encoder is used to extract the global features of the image, and the similarity is calculated and compared with the global features of all texts in the text library. The matching text with the greatest similarity is selected to complete the matching; For text retrieval images, a text encoder is used to extract global features of the text, and the similarity is calculated and compared with the global features of all images in the image library. The matching text with the greatest similarity is selected to complete the matching.

Images