A cross-modal image-text retrieval method and system based on adaptive contrast learning

Abstract

The application provides a cross-modal image-text retrieval method and system based on adaptive contrast learning, which comprises the following steps: introducing two boundary parameters in the probability distribution of each pair of positive samples, calculating the probability of image I corresponding to T, adjusting the semantic consistency of each pair of positive samples, and distinguishing the distribution characteristics of cloned negative samples; defining and calculating the saliency score of each batch of samples, selecting the negative sample sequence with the maximum and minimum saliency score as the observed cloned negative sample and the significant negative sample; according to the Gaussian discriminant analysis method, predicting all potential cloned negative samples of each batch of samples and generating anchor samples; according to the two boundary conditions of the similarity of the anchor samples, iteratively updating the two boundary parameters. The application is aimed at the cloned negative samples that are prone to appear in the image-text data set, and the Gaussian discriminant analysis is used to predict the potential cloned negative samples in each training batch without additional training. This lays a foundation for further learning of the high-dimensional key semantics of the cloned negative samples.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of cross-modal technology, and particularly relates to a cross-modal image-text retrieval method and system based on adaptive contrast learning. BACKGROUND

[0002] Cross-modal image-text retrieval has become one of the most fundamental multi-modal tasks in recent years, which includes searching images given a text query and retrieving corresponding sentences given an image. Existing methods for image-text matching usually learn a robust cross-modal fusion representation through various fusion paradigms (e.g., target-level or global-level alignment). Meanwhile, contrastive learning is widely used as the objective function for cross-modal tasks, which pulls the distance between similar positive samples closer in the latent space while expanding the distance between negative samples. Among these methods, Triplet Ranking Loss and Contrastive Loss have achieved remarkable success in image-text contrastive learning. Specifically, Triplet Ranking Loss maintains the relative distance between intra-batch samples by manually setting a fixed margin (see Faghri, Fartash, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. "VSE++: Improving visual-semantic embeddings with hard negatives." arXiv preprint arXiv:1707.05612 (2017).). Contrastive Loss is defined as the cross-entropy of the similarity between each pair of image-text samples after softmax normalization (see Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. "Learning transferable visual models from natural language supervision." In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.).In self-supervised learning and multi-modal tasks, InfoNCE and its variants are the main form of contrastive loss (see Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018). Cheng, Pengyu, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. "Club: A contrastive log-ratio upperbound of mutual information." In International conference on machine learning, pp. 1779-1788. PMLR, 2020.).

[0003] Despite the progress made, the current contrastive learning methods for cross-modal image-text retrieval face a serious challenge: during the training process, image-text pairs often contain semantically related text labels and similar visual clues, making them difficult to distinguish in practice (reflected in their very close similarity scores). There is no doubt that retrieving such samples is a suboptimal result, as the hand-labeled true labels exhibit more accurate and fine-grained semantics and tighter correspondence. Unlike false negatives, which should be considered as "positive samples", samples that are "semantically consistent but not optimal" are called clone negatives, which are extremely challenging in matching in such scenarios. It is extremely difficult to distinguish clone negatives through cross-modal fusion, as clone negatives are essentially "positive samples". Current image-text matching still mainly relies on basic triplet ranking loss or contrastive loss as the learning goal, which sets a fixed boundary during training, making it very difficult to dynamically handle clone negatives during training, as they have predefined labels for all image-text pairs (positive or negative samples). SUMMARY

[0004] In view of the defects in the prior art, the purpose of the present application is to provide a cross-modal image-text retrieval method and system based on adaptive contrastive learning.

[0005] According to a first aspect of the present application, there is provided a cross-modal image-text retrieval method based on adaptive contrastive learning, comprising:

[0006] using an image encoder and a text encoder extracting image I and text T visual modal feature v and text modal feature w respectively, and calculating the similarity to obtain the cosine similarity s of both in the same subspace;

[0007] Introducing two boundary parameters m1 and m2 in the probability distribution of each pair of positive samples, and combining the cosine similarity s, calculating the probability of the image I corresponding to the text T Adjusting the semantic consistency of each pair of positive samples, and distinguishing the distribution characteristics of cloned negative samples;

[0008] Defining and calculating the significance score of each batch of samples, and selecting the negative sample sequence with the maximum and minimum significance score in each batch of samples as the observed cloned negative sample S cln And the significant negative sample S sln ;

[0009] Based on the observed cloned negative sample S cln And the significant negative sample S sln , according to the Gaussian discriminant analysis method, predict all potential cloned negative samples of each batch of samples and generate anchor samples;

[0010] According to the two boundary conditions of the similarity of the anchor samples, iteratively update the values of the two boundary parameters m1 and m2;

[0011] After updating the values of the two boundary parameters m1 and m2, adaptively optimize the contrast learning loss to realize dynamic contrast learning between positive and negative samples, and train the image encoder And the text encoder

[0012] Using the trained image encoder And the text encoder Extract the visual modal feature and text modal feature of the image to be detected and the text to be detected, and calculate the similarity to obtain the cosine similarity of both in the same subspace, which is used as the basis for cross-modal image-text retrieval.

[0013] Preferably, the image encoder And the text encoder extract image I and text T visual modal feature v and text modal feature w respectively, and calculate the similarity to obtain the cosine similarity s of both in the same subspace, including:

[0014] The image encoder Convert the RGB pixels of the image I to feature representation v;

[0015] The text encoder Convert the text T to a word piece, and then convert the word piece to a text embedding w;

[0016] The cosine similarity of the feature representation v and the text embedding w is calculated to obtain s(I, T), which is used to measure the similarity of the image-text pair.

[0017] Preferably, two boundary parameters m1 and m2 are introduced into the probability distribution of each pair of positive samples, and the cosine similarity s is combined to calculate the probability p that the image I corresponds to the text T i (I), adjust the semantic consistency of each pair of positive samples, and distinguish the distribution characteristics of cloned negative samples, including:

[0018]

[0019] where i represents the index of the text T corresponding to the image I, i.e. (I, T i ) is a pair of image-text pairs, M represents the number of negative samples in a batch, m1 and m2 are two introduced boundary parameters, and the denominator is composed of a positive sample m1(s(I, T i )-m2) and M negative samples .

[0020] Preferably, the probability p that the image I corresponds to T i (I) is established to build an adaptive contrast learning objective function, specifically:

[0021]

[0022] where E I～D represents the expected value, which represents the expectation of the sum of all images I sampled from the data distribution D; H represents the cross-entropy loss function; N represents the number of samples in the batch; the subscript i represents the target label of the i-th sample, y i (I) is a one-hot encoding vector representing the real category of the image I or the text.

[0023] Preferably, the significance score of each batch sample is defined and calculated, and the negative sample sequence with the maximum and minimum significance score in each batch sample is selected as the observed cloned negative sample S cln and the significant negative sample S sln , including:

[0024] The significance score is defined as the difference between the average of the cosine similarity of the positive sample and the cosine similarity of the negative sample:

[0025]

[0026] s ii and s ij are cosine similarities, and s iis represents the positive sample similarity, i.e. the value along the main diagonal in the similarity matrix ij s represents the negative sample similarity, i.e. the value at coordinate (i,j) in the similarity matrix, i.e. the similarity between the i-th image or text sample and the j-th text or image sample;

[0027] The significance score of each batch sample is calculated according to the above definition;

[0028] The sample pair with the highest significance score is selected as the significant negative sample S sln , which contains 1 positive sample and M negative samples;

[0029] The sample pair with the lowest significance score is selected as the observed clone negative sample S cln , which contains 1 positive sample and M potential clone negative samples.

[0030] Preferably, the significance score is the updating standard of the boundary parameters m1 and m2, which is calculated and iterated in the process of each batch training.

[0031] Preferably, based on the observed clone negative sample S cln and the significant negative sample S sln , all potential clone negative samples of each batch sample are predicted and anchor samples are generated according to the Gaussian discriminant analysis method, including:

[0032] The class probability of each pair of image and text pair being a clone negative sample is calculated, and the distribution expression is:

[0033]

[0034] In the formula: a binary classification method is adopted, respectively represent whether the pair similarity is / is not a clone negative sample; s represents the cosine similarity score; represents the original score of the similarity without probability output, represents the original score of the similarity without probability output;

[0035] Based on the properties of Gaussian discriminant: Gaussian distribution with the same covariance of features, and similarity score as a single variable, the distribution expression of the class probability is simplified to a single variable distribution:

[0036] and wherein, represents a Gaussian distribution with mean u0 and variance ; represents a Gaussian distribution with mean μ1 and variance ;

[0037] The distribution expression is arranged as follows:

[0038]

[0039] wherein π1 and π0 are prior probabilities of and respectively; and given S sln and S cln are composed of the most representative significant negative samples and potential clone negative samples in a training batch, the respective negative samples are used to represent the empirical mean and variance μ1, μ0, σ1 and σ0;

[0040] The potential clone negative samples in each batch of samples are selected based on the following criteria:

[0041]

[0042] S * represents the predicted clone negative samples in each batch of samples;

[0043] The median sample in the set of S * is defined as an anchor sample.

[0044] Preferably, the two boundary conditions according to the similarity of the anchor sample are used to iteratively update the values of the two boundary parameters m1 and m2, including:

[0045] The probability that the anchor sample is correctly retrieved is calculated as:

[0046]

[0047] wherein anchor represents an anchor; I represents an image, T represents text, subscript u represents the index of an anchor sample, i.e. the index of text or image corresponding to the anchor image or text, and subscript k represents all indexes in a batch except u, i.e. the index of a negative sample of the anchor;

[0048] The two boundary conditions of the two boundary conditions are proposed, respectively as:

[0049]

[0050] is a constant; represents the sum of the logarithmic odds of all similarities except the index u.

[0051] In combination with the two boundary conditions, the updated m1 and m2 in each iteration process are derived.

[0052] ​Preferably, after updating the values of the two boundary parameters m1 and m2, the adaptive contrast learning loss is used to realize dynamic contrast learning between positive and negative samples in batch optimization to train the image encoder and the text encoder comprising:

[0053] obtaining the updated boundary parameters m1 and m2;

[0054] based on the updated boundary parameters m1 and m2, using the adaptive contrast learning objective function to calculate the loss, performing adaptive contrast learning to obtain the trained image encoder and the text encoder

[0055] According to a second aspect of the present application, a cross-modal image-text retrieval system based on adaptive contrast learning is provided, comprising:

[0056] a similarity module: using an image encoder and a text encoder extracting image I and text T visual modal features v and text modal features w respectively, and performing similarity calculation to obtain the cosine similarity s of both in the same subspace;

[0057] a boundary parameter module: introducing two boundary parameters m1 and m2 in the probability distribution of each pair of positive samples, and combining the cosine similarity s to calculate the probability of the image I corresponding to the text T adjusting the semantic consistency of each pair of positive samples, and distinguishing the distribution characteristics of the clone negative samples;

[0058] a saliency score module: defining and calculating the saliency score of each batch of samples, and selecting the negative sample sequence with the maximum and minimum saliency score in each batch of samples as the observed clone negative sample S cln and the significant negative sample S cln ;

[0059] an anchor module: based on the observed clone negative sample S cln and the significant negative sample S cln , according to the Gaussian discriminant analysis method, predicting all potential clone negative samples of each batch of samples and generating anchor samples;

[0060] a parameter updating module: according to the two boundary conditions of the similarity of the anchor samples, iteratively updating the values of the two boundary parameters m1 and m2;

[0061] a contrast learning module: after updating the values of the two boundary parameters m1 and m2, the adaptive contrast learning loss is used to realize dynamic contrast learning between positive and negative samples in batch optimization to train the image encoder and the text encoder

[0062] Inference module: using the trained image encoder and text encoder Extract the visual modal feature and the text modal feature of the to-be-detected image and the to-be-detected text, and perform similarity calculation to obtain the cosine similarity of both in the same subspace as the basis for cross-modal image-text retrieval.

[0063] Compared with the prior art, the embodiment of the present application has at least one of the following beneficial effects:

[0064] The cross-modal image-text retrieval method and system based on adaptive contrast learning in the embodiment of the present application predict the potential cloned negative samples in each training batch without additional training by using Gaussian discriminant analysis, laying a foundation for further learning of the high-dimensional key semantics of the cloned negative samples.

[0065] The cross-modal image-text retrieval method and system based on adaptive contrast learning in the embodiment of the present application jointly improve the traditional contrast learning method, invent a method based on adaptive contrast learning, propagate the semantics of the cloned negative samples in the training process through two dynamically fine-tuned boundary parameters, enhance the closeness of the positive samples, and effectively solve the influence of the cloned negative samples. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 The flowchart of the cross-modal image-text retrieval method based on adaptive contrast learning in an embodiment of the present application;

[0067] Figure 2 The comparison of the cloned negative sample, the observed positive sample and the significant negative sample in an embodiment of the present application;

[0068] Figure 3 The sample distinction degree in the joint subspace in an embodiment of the present application;

[0069] Figure 4 The framework structure diagram of the cross-modal image-text retrieval method based on adaptive contrast learning in an embodiment of the present application;

[0070] Figure 5 The final result diagram of the image-text retrieval based on adaptive contrast learning for a pair of cloned negative samples in a specific embodiment of the present application;

[0071] Figure 6 The visualization result diagram of the training process of the adaptive boundary parameter introduced by the network in a specific embodiment of the present application;

[0072] Figure 7A visual text joint feature clustering visualization result schematic diagram in an embodiment of the present application;

[0073] Figure 8 A comparison result of final retrieval accuracies of different retrieval methods in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The embodiments of the present application will be described in detail below with reference to the accompanying drawings: The embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation manners and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0075] In an embodiment of the present application, a cross-modal image-text retrieval method based on adaptive contrast learning is provided, as shown in Figure 1 The method comprises the following steps:

[0076] S100: image and text visual modal features v and text modal features w are extracted respectively by using an image encoder and a text encoder , and a similarity calculation is performed to obtain a cosine similarity s of the two in the same subspace;

[0077] S200: two boundary parameters m1 and m2 are introduced in the probability distribution of each pair of positive samples, and the cosine similarity s is combined to calculate the probability P of the image I corresponding to the text T The semantic consistency of each pair of positive samples is adjusted, and the distribution characteristics of the cloned negative samples are distinguished;

[0078] S300: the saliency score of each batch of samples is defined and calculated, and the negative sample sequence with the maximum and minimum saliency score in each batch of samples is selected as the observed cloned negative sample S cln and the salient negative sample S sln ;

[0079] S400: based on the observed cloned negative sample S cln and the salient negative sample S sln , according to the Gaussian discriminant analysis method, all potential cloned negative samples of each batch of samples are predicted and anchor samples are generated;

[0080] S500: according to the two boundary conditions of the similarity of the anchor samples, the values of the two boundary parameters m1 and m2 are iteratively updated;

[0081] S600, after updating the values of the two boundary parameters m1 and m2, the adaptive contrast learning loss is optimized batch by batch to realize dynamic contrast learning between positive and negative samples, and the image encoder and the text encoder

[0082] S700, using the trained image encoder of S100-S600 and the text encoder The visual modal feature and the text modal feature of the image to be detected and the text to be detected are extracted, and similarity calculation is performed to obtain the cosine similarity of both in the same subspace as the basis for cross-modal image-text retrieval.

[0083] The above embodiment is used to solve the problems of uncertainty of matching results and insufficient optimization caused by cloning negative samples in the prior art. By introducing two adaptive boundary parameters (scaling parameter m1 and offset parameter m2), the closeness of the positive sample and the distinguishability of the cloned negative sample are dynamically adjusted, so that the precision and robustness of image-text matching are effectively improved; the anchor sample is determined by the Gaussian discriminant analysis method without additional training, so that the influence of the cloned negative sample on the training process is dynamically adjusted to optimize the training process.

[0084] In addition, it can be further applied to a weakly supervised image-text retrieval scene, and the pseudo-labeled description generated by the image is used to replace manual labeling, so that the retrieval performance of the system on a low-quality image-text dataset is improved, laying a foundation for large-scale image-text contrast learning, and having universal applicability.

[0085] In one preferred embodiment of the application, step S100 is implemented, and based on the image-text pair, image and text features are extracted by using image and text encoders and the similarity is calculated. Specifically:

[0086] S101, the image encoder converts the RGB pixels of the input image into a high-dimensional feature representation v.

[0087] S102, the text encoder converts the text into a word piece and then converts it into a high-dimensional text embedding w.

[0088] S103, the cosine similarity of the feature representations v and w of the image I and the text T is calculated to obtain s(I, T), which is used to measure the similarity of the image-text pair.

[0089] Of course, the order of S101 and S102 is not unique, and in other embodiments, the implementation steps can be S102 / S101 or simultaneous implementation.

[0090] In the training process, cloned negative samples and significant negative sample queues need to be selected from each training batch, and anchor samples are obtained. In one preferred embodiment of the application, S200\S300\S400 are sequentially implemented to obtain target anchor samples. Specifically:

[0091] S200, two boundary parameters m1 and m2 are introduced in the probability distribution of each pair of positive samples, and the cosine similarity s is combined to calculate the probability of image I corresponding to T Adjust the semantic consistency of each pair of positive samples, and distinguish the distribution characteristics of the cloned negative samples. The specific implementation process is as follows:

[0092] S201, the similarity p(I) passing through the softmax transfer function can be expressed as:

[0093]

[0094] Where m1 and m2 are two introduced boundary parameters for adaptively adjusting the semantic consistency of the positive sample image-text pair and distinguishing the distribution characteristics of the cloned negative sample pair.

[0095] The denominator is composed of M+1 samples, specifically: one positive sample and M negative samples. If the momentum storage is not used, M is equal to the training batch size N in value. τ is a temperature parameter for controlling the overall supervision information. p i (I) represents the probability of image I corresponding to T.

[0096] Further, the adaptive contrast learning objective function can be expressed as:

[0097]

[0098] Where y(I)y i (I) is a one-hot encoding vector representing the true class (or text) of image I.

[0099] The distance between the positive sample image-text pair is made closer, while the distance between the predefined negative sample pair is made further. E I～D represents the expected value, representing the expectation of summing all images I sampled from the data distribution D; H represents the cross-entropy loss function, which measures the difference between the model's predicted probability distribution p(I) and the target label y(I); N represents the number of samples in the batch; The subscript i represents the target label of the ith sample.

[0100] In some specific embodiments, if the momentum storage is not used, M is equal to the training batch size N in value. If the momentum storage is used, the value of M can be set by a hyperparameter, such as setting M to 4096 to cover as many negative samples as possible.

[0101] The above embodiments, in particular, introduce two boundary parameters as scaling and offset factors, for synchronously enhancing the closeness of positive samples and containing supervision on cloned negative samples.

[0102] S300, define and calculate the saliency score of each batch sample, and select the negative sample sequence with the maximum and minimum saliency score in each batch sample as the observed cloned negative sample S cln and the salient negative sample Ssln The specific process is as follows:

[0103] S301, in order to predict the potential clone negative samples in each batch of image-text pairs, first define the significance score, which can be expressed as:

[0104]

[0105] s ii And s ij are cosine similarity, s ii represent the positive sample similarity, that is, the value along the main diagonal in the similarity matrix, s ij represent the negative sample similarity, that is, the value of the coordinate (i, j) in the similarity matrix, that is, the similarity between the ith image or text sample and the jth text or image sample;

[0106] The significance score better reflects the degree of significance by considering the similarity of negative sample pairs.

[0107] S302, the sample pair with the highest significance score is taken as the significant negative sample S sln , which contains 1 positive sample and M significant negative samples. The sample pair with the lowest significance score is defined as the observed clone negative sample S cln , which contains 1 positive sample and M negative samples;

[0108] Figure 2 It shows the difference between clone negative samples and commonly seen significant negative samples. Specifically, image-text pairs often contain semantically related text labels and similar visual clues, making them difficult to distinguish in practice (reflected in their very close similarity scores). There is no doubt that retrieving such samples is a suboptimal result, because the hand-labeled true labels exhibit more accurate and fine-grained semantics, and the corresponding relationship is closer.

[0109] S400, based on the observed clone negative sample S cln and the significant negative sample S sln , according to the Gaussian discriminant analysis method, all potential clone negative samples of each batch of samples are predicted and anchor samples are generated. The specific process is as follows:

[0110] S401, first from the class probability, which can be expressed as:

[0111]

[0112] In the formula: adopt binary classification method, respectively represent whether the pair similarity is / is not a clone negative sample; s represents the cosine similarity score; represent the original score of the similarity without probability output, The raw score represents the similarity without the probability output. A clone-negative sample predictor without additional training can be obtained by analyzing the sample distribution of clone-negative samples and Gaussian discriminant analysis.

[0113] S402, in Gaussian discriminant analysis, features are usually assumed to follow Gaussian distribution with the same covariance. Since the similarity score is a single-dimensional variable, the above distribution expression can be simplified as a single-variable distribution. Wherein And Combined with this assumption and We can get:

[0114]

[0115] Wherein, π1 and π0 are the prior probabilities of And S sln And S cln Composed of the most representative significant negative samples and potential clone-negative samples in a training batch, the respective negative samples are used to represent the empirical mean and variance μ1, μ0, σ1 and σ0;

[0116] S403: The selection of potential clone-negative samples in small batches is based on the following criteria:

[0117]

[0118] S * Represents the predicted clone-negative sample in each batch sample.

[0119] S404, the anchor point is defined as the median sample in S * Set, thereby obtaining the anchor sample.

[0120] In the above embodiment, in order to adjust the boundary parameters step by step, based on Gaussian discriminant analysis, an anchor sample is selected from the similarity score of each batch without introducing additional training. The anchor sample can effectively reflect the strength of the clone-negative sample in the batch, and impose a penalty through the boundary parameter, thereby adaptively expanding the distance between the positive sample and the clone-negative sample.

[0121] In a preferred embodiment of the present application, step S500 is implemented, and the values of the two boundary parameters m1 and m2 are iteratively updated according to the two boundary conditions of the similarity of the anchor sample, and the specific process is as follows:

[0122] S501, calculate the anchor probability.

[0123] To calculate the boundary parameters m1 and m2 introduced in step S2 in each batch learning, the similarity p(I) passing through the softmax transfer function is specifically analyzed. First, the probability corresponding to the anchor point (i.e., the probability of the correct matching of the anchor point image-text pair) can be calculated by the anchor point obtained in the fourth step of the equation:

[0124]

[0125] wherein, anchor represents an anchor point; I represents an image, T represents text, subscript u represents the index of the anchor point sample, i.e., the index of the text or image corresponding to the anchor point image or text, and subscript k represents all indexes in the batch except u, i.e., the index of the negative sample of the anchor point;

[0126] S502, propose boundary conditions.

[0127] Since the anchor point represents the set average probability of the potential cloned negative sample in each batch, the corresponding to a large extent reflects the degree of approximate convergence of the model. Assuming that the probability can be controlled, the overall supervision in each batch can be well regulated. Therefore, the m1 and m2 scheme can be adjusted based on The boundary conditions of are analyzed: since the cross-entropy loss is expressed as When the similarity score s(I_u, T_u) of the anchor point is constantly approaching 1 (indicating high similarity), a relatively small m1 will unnecessarily punish the loss of the correctly matched image-text pair. Therefore, when s(I_u, T_u) is close to 1, should also be as close to 1 as possible to improve the ability to distinguish cloned negative samples. This condition can be expressed as:

[0128]

[0129] In combination with the expression of , another boundary condition is:

[0130]

[0131] ∈ is a constant for avoiding invalid calculation when m1 reaches 0; represents the sum of the logarithmic probabilities of all similarities except the index u.

[0132] S503, derive new m1 and m2.

[0133] With the two boundary conditions, the updated m1 and m2 in each iteration process can be derived and adaptively update the contrastive learning loss during the training process. In this way, The semantics of the cloned negative samples can be effectively propagated through m1 and m2.

[0134] The above embodiments consider and handle the cloned negative samples in the dataset, and effectively promote the model to distinguish them during the training process through an adaptive image-text contrastive learning method.

[0135] Based on the same inventive concept, in other embodiments of the present application, an adaptive contrast learning-based cross-modal image-text retrieval system is provided, comprising:

[0136] Similarity module: using image encoder and text encoder extract image I and text T visual modal features v and text modal features w respectively, and calculate the cosine similarity s of both in the same subspace;

[0137] Boundary parameter module: introduce two boundary parameters m1 and m2 in the probability distribution of each pair of positive samples, and combine the cosine similarity s to calculate the probability of image I corresponding to text T Adjust the semantic consistency of each pair of positive samples, and distinguish the distribution characteristics of cloned negative samples;

[0138] Significance score module: define and calculate the significance score of each batch of samples, and select the negative sample sequence with the maximum and minimum significance score in each batch of samples as the observed cloned negative sample S cln and significant negative sample S cln ;

[0139] Anchor module: based on the observed cloned negative sample S cln and significant negative sample S sln , according to the Gaussian discriminant analysis method, predict all potential cloned negative samples of each batch of samples and generate anchor samples;

[0140] Parameter update module: according to the two boundary conditions of the similarity of the anchor samples, iteratively update the values of the two boundary parameters m1 and m2;

[0141] Contrastive learning module: after updating the values of the two boundary parameters m1 and m2, adaptively update the contrastive learning loss, and implement dynamic contrastive learning between positive and negative samples in batches to train the image encoder and text encoder

[0142] Inference module: using the trained image encoder and text encoder The visual modal feature and the text modal feature of the to-be-detected image and the to-be-detected text are extracted, and a cosine similarity in a same subspace is calculated as a basis for cross-modal image-text retrieval.

[0143] The modules / units in the above examples can refer to the implementation techniques of the steps of the cross-modal image-text retrieval method based on adaptive contrast learning in the above embodiments, and details are not described herein.

[0144] To verify the feasibility and effectiveness of the above embodiments, in one specific embodiment of the present application, the image frames are from the image-text pairs in the database Flickr30K for cross-modal image-text retrieval performance evaluation. Flickr30K contains 31,783 images, each corresponding to 5 different sentences. According to the manner of 29,783 / 1,000 / 1,000, it is divided into a training set / a test set / a validation set. MS-COCO contains 123,287 images, each corresponding to 5 sentences. It is divided into a training set / a test set / a validation set according to 113,287 / 5,000 / 5,000 images. The test set is further divided into MS-COCO 1K (average result of 5 test sets) and MS-COCO 5K (result of 5,000 test images).

[0145] Figure 3 It is shown that the distinguishing distance between negative samples using the adaptive contrast learning method in the above embodiments is more significant than the widely used triplet ranking loss and contrast loss, which confirms its effectiveness.

[0146] Figure 4 The composition of the cross-modal image-text retrieval network structure is shown. Specifically, the salient negative samples and the observed cloned negative samples are selected by the saliency score. Then, based on the Gaussian discriminant analysis, two boundary parameters introduced in the loss function are dynamically adjusted, so as to achieve a more fine contrast learning goal. The adaptive contrast learning method effectively adjusts the distance between the positive sample and the cloned negative sample by gradually adjusting the boundary parameters. Unlike the traditional contrast learning method, this network structure can adaptively adjust the parameters according to the strength of the cloned negative sample, thereby enhancing the robustness in challenging scenarios.

[0147] In summary, the method of the present embodiment dynamically adjusts the closeness of the positive sample and the distinguishability of the cloned negative sample by introducing two adaptive boundary parameters (scaling parameter and offset parameter), thereby effectively improving the accuracy and robustness of image-text matching; the anchor sample is determined by the Gaussian discriminant analysis method without additional training, so that the influence of the cloned negative sample on the training process can be dynamically adjusted to optimize the training process.

[0148] As Figure 5The Rank@K is used as the evaluation index (see Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pp. 201-216, 2018.). Specifically, given a query image / text, the database images / texts are ranked according to their similarity to the query image / text. If at least one correct image / text appears in the top K positions of the ranking, the retrieval is considered correct, otherwise it is considered incorrect. Figure 5 The effect of the adaptive contrast learning method in the embodiment of the application on processing the cloned negative sample is analyzed, the matching results are ranked according to the similarity scores, and the adaptive contrast learning loss is compared with two other widely used objective functions, and the following conclusions are revealed: compared with the contrast loss and the triplet ranking loss, the adaptive contrast learning loss performs better in R@1, and can successfully exclude other potential negative samples, proving that the effect of the embodiment of the application on processing the cloned negative sample is significant; at the same time, the cross-modal fusion representation learned by the model under the adaptive contrast learning loss performs better in identifying the cloned negative sample. For example, for the cloned negative sample of “a group of people running or participating in a marathon in the city”, the similarity score is only 0.30. In contrast, although the model retrieves the true label under the contrast loss and the triplet ranking loss, the similarity score of the cloned negative sample is still a suboptimal solution. The retrieval method based on adaptive contrast learning has better discrimination effect on more challenging samples.

[0149] Figure 6 The convergence process of the boundary parameters introduced in the application in the model training. m2 gradually increases and stabilizes at about 0.36, while m1 stabilizes at about 38.1 after experiencing a certain shock at the 10th iteration period, showing excellent convergence speed.

[0150] Figure 7 The distribution of the visual-text joint embedding in the t-SNE space at the 10th iteration period is shown. Compared with the contrast loss and the triplet ranking loss, the adaptive contrast loss shows better clustering performance, further proving that it effectively accelerates the training convergence process of the cross-modal image-text retrieval model.

[0151] Figure 8is the comparison result of the final search accuracy based on the performance obtained by the embodiment. Two data sets, Flickr30K and MS-COCO, are included. A variety of competitive cross-modal image-text retrieval methods are selected as baseline models, and the plug-and-play effect of the image-text retrieval method based on adaptive contrast learning (AdaCL) is verified. It can be observed that the embodiment is superior to the original baseline method on each backbone network type. On the Flickr30K and MS-COCO data sets, the (R@1, R@5, R@10) of the embodiment in the image-text retrieval task all achieve very significant absolute improvement, indicating that the embodiment can be applied to more general network architectures without additional design of cross-modal fusion mechanisms.

[0152] In summary, the cross-modal image-text retrieval method based on adaptive contrast learning provided by the embodiment considers and solves the problems of matching result uncertainty and optimization deficiency caused by cloned negative samples, reduces the influence and challenges of cross-modal image-text retrieval models. The adaptive contrast learning is invented, the closeness between the image-text pairs is dynamically adjusted by introducing two adjustable boundary parameters and an anchor point, and the semantic information of the potential cloned negative samples is learned; the anchor point is selected according to the distribution of the negative samples, without explicit training, so as to realize gradual adjustment and enhance the supervision effect in small batches; the adaptive contrast learning method shows excellent robustness, and the superior performance shows the great potential of adaptive contrast learning in reducing the dependence on artificial annotation.

[0153] Although the content of the present application has been described in detail through the above preferred embodiments, it should be recognized that the above description should not be considered as limiting the present application. After reading the above content, various modifications and alternatives of the present application will be apparent to those skilled in the art. Therefore, the scope of protection of the present application should be defined by the appended claims.

Claims

1. A cross-modal image-text retrieval method based on adaptive contrast learning, characterized in that, Comprising: Utilizing an image encoder and a text encoder The image I and the text T visual modal feature v and the text modal feature w are extracted respectively, and the similarity calculation is performed to obtain the cosine similarity s of both in the same subspace. Introducing two boundary parameters m1 and m2 in the probability distribution of each pair of positive samples, and combining the cosine similarity s, the probability that the image I corresponds to the text T is calculated Adjusting the semantic consistency of each pair of positive samples, and distinguishing the distribution characteristics of cloned negative samples; The significance score of each batch sample is defined and calculated, and the negative sample sequence with the maximum and minimum significance score in each batch sample is selected as the observed clonal negative sample S cln and the significant negative sample S sln ; Based on the observed clone-negative samples S cln and the significantly negative samples S sln , according to the Gaussian discriminant analysis method, predict all potential clone-negative samples of each batch sample and generate anchor samples; According to the similarity of the anchor sample, two boundary conditions are used to iteratively update the values of the two boundary parameters m1 and m2; After updating the values of the two boundary parameters m1 and m2, the adaptive contrast learning loss is used to realize dynamic contrast learning between positive and negative samples in batches to train the image encoder and the text encoder using the trained image encoder and the text encoder extract visual modality features and text modality features of the image to be detected and the text to be detected, and perform similarity calculation to obtain the cosine similarity of both in the same subspace as the basis for cross-modal image-text retrieval.

2. The cross-modal text-image retrieval method based on adaptive contrast learning according to claim 1, characterized in that, The image encoder and the text encoder extract visual modality features v and text modality features w of the image I and the text T respectively, and perform similarity calculation to obtain cosine similarity s of both in the same subspace, including: The image encoder converting the RGB pixels of the image I into a feature representation v; The text encoder converts the text T into tokens, and converts the tokens into text embeddings w; The cosine similarity between the feature representation v and the text embedding w is calculated to obtain s(I, T), which is used to measure the similarity of the image-text pair.

3. The cross-modal graphic-text retrieval method based on adaptive contrast learning according to claim 1, characterized in that, introducing two boundary parameters m1 and m2 in the probability distribution of each pair of positive samples, and combining the cosine similarity s, calculating the probability p of the image I corresponding to the text T i (I), adjusting the semantic consistency of each pair of positive samples, and distinguishing the distribution characteristics of cloned negative samples, comprising: In the formula, i represents the subscript index of the text T corresponding to the image I, that is, (I, T i ) is a pair of image-text pairs, M represents the number of negative samples in a batch, m1 and m2 are two introduced boundary parameters, and the denominator is composed of a positive sample m1(s(I, T i )-m2) and M negative samples .

4. The cross-modal text-image retrieval method based on adaptive contrast learning according to claim 3, characterized in that, a probability p that the image I corresponds to T i (I), an adaptive contrast learning objective function is established, specifically: where E I～D represents the expected value of the sum over all images I sampled from the data distribution D; H represents the cross-entropy loss function; N represents the number of samples in the batch; the index i represents the target label of the ith sample, y i (I) is a one-hot encoded vector representing the true class of image I.

5. The cross-modal text-image retrieval method based on adaptive contrast learning according to claim 1, characterized in that, The definition and calculation of the significance score of each batch sample, and the selection of the negative sample sequence with the maximum and minimum significance score in each batch sample as the observed clone negative sample S cln and the significant negative sample S sln , comprising: The significance score is defined as the difference between the average cosine similarity of positive samples and the average cosine similarity of negative samples: s ii and s ij are cosine similarities, s ii represents positive sample similarity, i.e., the value along the main diagonal in the similarity matrix, s ij represents negative sample similarity, i.e., the value at coordinate (i,j) in the similarity matrix, i.e., the similarity of the ith image or text sample to the jth text or image sample; According to the above definition, the significance score of each batch of samples is calculated; Selecting the sample pair with the highest saliency score as the significant negative sample S sln wherein there are 1 positive sample and M negative samples; Select the sample pair with the lowest significance score as the observed clonal negative sample S cln containing 1 positive sample and M potential clonal negative samples.

6. The cross-modal text-image retrieval method based on adaptive contrast learning according to claim 5, characterized in that, The significance score is the update standard of the boundary parameters m1 and m2, which is calculated and iterated in the process of each batch training.

7. The cross-modal image-text retrieval method based on adaptive contrast learning according to claim 1, characterized in that, The observation clone-negative sample S cln and the significant negative sample S sln According to the Gaussian discriminant analysis method, all potential clone-negative samples of each batch sample are predicted and anchor samples are generated, comprising: The class probability of each pair of image and text pair being a cloned negative sample is calculated, and its distribution expression is: where: binary classification method is used, represents pair-wise similarity is / is not a negative sample; s represents cosine similarity score; represents the original score of similarity without probability output, represents the original score of similarity without probability output; Based on the properties of Gaussian discrimination: Gaussian distribution with the same covariance of features, and similarity score as a single variable, the distribution expression of the class probability is simplified to a single variable distribution: and wherein, represents a Gaussian distribution with mean u0and variance σ0, represents a Gaussian distribution with mean μ1and variance σ1. The distribution expression is arranged as: where π1and π0are the prior probabilities of and respectively; given S sln and S cln consist of the most representative significant negative samples and potential clonal negative samples within a training batch, respectively, and are used to represent the empirical mean and variance μ1, μ0, σ1, and σ0, respectively; Based on the following criteria, the potential cloned negative samples in each batch of samples are selected: S * represents the predicted clonal negative samples in each batch sample; The S is defined as * The median sample in the set is the anchor sample.

8. The cross-modal text-image retrieval method based on adaptive contrast learning according to claim 1, characterized in that, According to the similarity of the anchor sample, two boundary conditions are used to iteratively update the values of the two boundary parameters m1 and m2, comprising: The probability that the anchor sample is correctly retrieved is calculated: wherein, anchor denotes an anchor; I denotes an image, T denotes text, the subscript u represents the index of the anchor sample, i.e. the index of the text or image corresponding to the anchor image or text, and the subscript k represents all indices in the batch except u, i.e. the negative sample indices of the anchor; The two boundary conditions of the proposed are respectively: ∈ is a constant; represents the sum of all similarity log-likelihoods except for index u; Combined with the two boundary conditions, the updated m1 and m2 in each iteration process are derived.

9. The cross-modal text-image retrieval method based on adaptive contrast learning according to claim 4, characterized in that, The adaptive contrast learning loss is batch-optimized to realize dynamic contrast learning between positive and negative samples after updating the values of the two boundary parameters m1 and m2 and the text encoder comprises: The updated boundary parameters m1 and m2 are obtained; Based on the updated boundary parameters m1 and m2, a loss is calculated by using the adaptive contrast learning objective function, adaptive contrast learning is performed, and a trained image encoder is obtained and a text encoder 10. A cross-modal image-text retrieval system based on adaptive contrastive learning, characterized in that, Comprising: Similarity module: using image encoder and text encoder Extract image I and text T visual modal features v and text modal features w respectively, and perform similarity calculation to obtain cosine similarity s of both in the same subspace; a boundary parameter module: introducing two boundary parameters m1 and m2 in the probability distribution of each pair of positive samples, and combining the cosine similarity s to calculate the probability that the image I corresponds to the text T adjusting the semantic consistency of each pair of positive samples, and distinguishing the distribution characteristics of cloned negative samples; The significance score module defines and calculates the significance score of each batch sample, and selects the negative sample sequence with the maximum and minimum significance score in each batch sample as the observed clonal negative sample S cln and the significant negative sample S sln ; Anchors module: based on the observed clonal negatives S cln and the significant negatives S sln , predict all potential clonal negatives for each batch of samples and generate anchors according to Gaussian Discriminant Analysis method; Parameter update module: according to the similarity of the anchor sample, two boundary conditions are used to iteratively update the values of the two boundary parameters m1 and m2; Contrastive learning module: after updating the values of the two boundary parameters m1 and m2, the adaptive contrastive learning loss is realized to optimize the dynamic contrastive learning between positive and negative samples in batches, and the image encoder is trained and the text encoder Inference module: using the trained image encoder and the text encoder Extract visual modal features and text modal features of the image to be detected and the text to be detected, and perform similarity calculation to obtain the cosine similarity of both in the same subspace as the basis for cross-modal image-text retrieval.

Images