Cross-modal retrieval method based on global-local comparative learning

Abstract

The invention provides a cross-modal retrieval method based on global-local comparative learning, which comprises the following steps of: enabling an instance to correspond to a plurality of positive samples by adopting a scoring mechanism, inputting global features of an image and a text into a variational auto-encoder, and adding generated features into a positive sample set for learning, so as to further enhance the discrimination capability of a model; a ResNet-152 network and a BERT model are adopted to extract local features of an image and a text respectively, a soft attention mechanism is adopted to learn related information between image areas and text words, and the soft attention mechanism calculates correlation between each text word and each image area through weighting; therefore, the text words and the image areas do not need to be subjected to fine-grained labeling. The problem that positive and negative samples are not flexibly selected in a current comparative learning method is solved, and the problem that detail semantic information among data is ignored in an existing method is solved.

Description

Technical Field

[0001] The present invention relates to the field of retrieval, and in particular to a cross-modal retrieval method. Background Art

[0002] Compared with supervised methods, unsupervised cross-modal retrieval methods do not need to use semantic labels, which saves manpower and material resources when labeling labels, and can avoid the potential problem of inaccurate labeling. Most of the existing unsupervised cross-modal retrieval methods are based on contrastive learning, but they still have the following problems: ① The selection of positive and negative samples is not flexible enough, resulting in semantically similar data being far apart in the common subspace, which reduces the model's discrimination performance. ② Only focusing on the global correlation between images and texts, ignoring the impact of detailed semantic information between data on the model. The purpose of the present invention is to propose a cross-modal retrieval method based on global-local contrastive learning, design a new scoring mechanism, and use a soft attention mechanism to learn the detailed semantic information between images and texts to solve the above problems.

[0003] The image-text cross-modal retrieval task refers to retrieving text data with similar semantics given an image, or retrieving image data with similar semantics given a text. This task performs retrieval by exploring the semantically related information between the image and the text. However, since data of different modalities are distributed differently in the feature space and their expressions are also quite different, it is impossible to directly compare similarities. In this regard, most existing image-text cross-modal retrieval methods map data of different modalities to a common subspace, and complete cross-modal retrieval by comparing the similarities of data of different modalities in this space. Based on the above analysis, it is not difficult to see that in order to accurately complete the cross-modal retrieval task, the following key issues need to be resolved:

[0004] ① How to extract features from raw data of different modes and map them into an effective common subspace;

[0006] ② How to place semantically similar data in close proximity and semantically dissimilar data in far distances in the common subspace.

[0007] For the first problem, a common approach is to construct a common subspace, then use a deep neural network to extract the semantic features of the original data (images and text) and map it to a point in the space, where each point has a specific meaning and corresponds to the semantic information of the real world. It should be noted that since image and text data have different characteristics, the differences between different modalities and their application scenarios should be considered when selecting a feature extraction network, so as to fully extract the semantic features of the original data.

[0008] In the image-text cross-modal retrieval dataset, images and texts usually appear in pairs, that is, one image corresponds to one (paragraph) of text. Most of the existing image-text cross-modal retrieval methods improve the retrieval accuracy by retaining the similarity information between paired data when training the dataset. However, due to the large scale of image-text datasets, there may be multiple texts with similar semantics for one image, and conversely, there may be multiple semantically similar images for one text. Considering only the similarity information between paired data may cause semantically similar unpaired data to be far apart in the common subspace, which is obviously not conducive to improving the discriminative performance of the model. Therefore, the second key problem in image-text cross-modal retrieval is how to reduce the distance between semantically similar data in the common subspace. To address this problem, retaining the similarity information within and between modalities can strengthen the association between semantically similar data, improve the model's discriminative ability, and thus improve the model's retrieval accuracy.

[0009] In the existing image-text cross-modal retrieval method based on contrastive learning, paired images and texts are usually selected as positive samples of each other, that is, it is assumed that an image is anchor data, and its positive sample has only one text corresponding to it. Then, the distance of positive samples in the common subspace is reduced by minimizing the loss function, and the distance of negative samples in the common subspace is expanded to make the model discriminative. However, since the image-text data set is usually large in scale, there may be multiple texts with similar semantics to a picture, and there may be multiple pictures with similar semantics to a text. Therefore, the first problem that the invention intends to solve is that the existing image-text cross-modal retrieval method based on contrastive learning has the problem that the selection of positive and negative samples is not flexible enough, resulting in weak model discrimination ability, that is, there are still many data with similar semantics to the anchor data in the data set that are mistakenly regarded as negative samples for learning, resulting in semantically similar data being far away in the common subspace, thereby reducing the model's discrimination ability. In addition, when using contrastive learning to solve the cross-modal retrieval problem, the existing method only focuses on the global correlation between image and text, ignoring the local correlation information between image areas and text words, which has a negative impact on the model retrieval accuracy. Therefore, the second problem that the present invention intends to solve is how to make the model focus on the detailed semantic-related information of multimodal data on the basis of paying attention to the global relevance of multimodal data, so as to further improve the performance of cross-modal retrieval. Summary of the invention

[0010] In order to overcome the deficiencies of the prior art, the present invention provides a cross-modal retrieval method based on global-local contrastive learning.

[0011] The specific steps of the technical solution adopted by the present invention to solve its technical problem are:

[0012] Step 1: Global Contrastive Learning

[0013] A scoring mechanism is used to make one instance correspond to multiple positive samples. The global features of the image and text are input into the variational autoencoder, and the generated features are added to the positive sample set for learning, thereby further enhancing the model's discriminative ability.

[0014] (1) Flexible scoring mechanism for selecting positive and negative samples

[0015] A scoring mechanism is used to flexibly select positive and negative samples. First, m sample pairs are randomly selected from the data set to construct a batch M. Sample pairs s i and sample pairs s j The similarity score Sc between ij for:

[0016] Sc ij =min(cos(v i ,v j ),cos(t i ,t j )) (1)

[0017] In formula (1), cos(a,b)=(a·b) / (||a||2·||b||2) represents the cosine similarity between two samples. The similarity score Sc is high only when the cosine similarity between the image and the text of the two sample pairs is high. ij Take the larger value, the sample pair s i Include image v i and text t i , then, according to the similarity score Sc ij For each sample pair s i Select positive and negative samples, s i The positive sample set is P i , P i Contains the current batch with s i The top K sample pairs with the highest similarity scores; definition s i The negative sample set is N i , N i P in the current batch i The complement of

[0018] P i =TopK(Sc ij ),j=1,2,...,m (2)

[0019]

[0020] In formula (3), The symbol for the complement set is used. Through the scoring mechanism, K sample pairs with higher similarity scores are selected as positive samples of the anchor data, and the remaining sample pairs are used as negative samples of the anchor data, thereby realizing the flexible selection of positive and negative samples.

[0021] (2) Global inter-modality contrastive learning

[0022] Global inter-modal contrastive learning aims to reduce the distance between paired images and texts in the common subspace and complete global semantic alignment. In inter-modal contrastive learning, the distance between two samples in the same sample pair is reduced, and the distance between two samples in different sample pairs is expanded. Normalized temperature scaled cross entropy loss is used as the loss function. The numerator of the loss function is the cosine similarity of the positive sample pair, and the denominator of the loss function is the cosine similarity between the negative sample pairs. By minimizing the loss function, the cosine similarity between the positive sample pairs can be increased, and the cosine similarity between the negative samples can be reduced, thereby reducing the distance between the positive samples and expanding the distance between the negative samples in the common subspace. The image-text global inter-modal contrastive loss is defined as for:

[0023]

[0024] Similarly, the text-image global inter-modality contrast loss is defined as:

[0025]

[0026] In formulas (4) and (5), τ represents the temperature hyperparameter. For each image v in the current batch M, i and text t i For contrastive learning, the global inter-modal contrast loss is defined as:

[0027]

[0028] By minimizing the loss function in formula (6), the distance between image-text pairs in the common subspace can be reduced, thereby achieving global semantic alignment;

[0029] (3) Global intra-modal contrastive learning

[0030] Global intra-modal contrastive learning aims to map semantically similar data to similar positions in the common subspace, thereby improving the model's discriminative ability;

[0031] First, the scoring mechanism is used to select the top K sample pairs with the highest similarity scores to construct a positive sample set. Then, in order to enable the anchor samples to learn positive samples with higher similarity during intra-modal contrast learning and further improve the model's discrimination ability, the initial image v is converted into i and text t i Input the variational self-encoder to get the generated image and generate text And will generate the image and generate text Add to the positive sample set; new positive sample set P i ' is expressed as:

[0032]

[0033] In formula (7), is a pair of generated samples consisting of a generated image and a generated text. The global intra-modal contrastive learning uses the normalized temperature-scaled cross entropy loss as the loss function. The image-image global intra-modal contrastive loss is defined as:

[0034]

[0035] Similarly, the text-text global intra-modality contrastive loss is defined as:

[0036]

[0037] In formulas (8) and (9), K+1 is the number of positive samples selected by the scoring mechanism, and 1 means there is one positive sample generated by the variational autoencoder. For each image v in the current batch, i and text t i For contrastive learning, the global intra-modal contrastive loss is defined as:

[0038]

[0039] By minimizing the loss function in formula (10), the distance between the anchor data and the positive sample set in the feature subspace can be reduced, and the distance between the anchor data and the negative sample set in the feature subspace can be expanded, thereby improving the discriminative performance of the model. The global contrast loss is defined as:

[0040] L Global =αL GInter +βL GIntra (11)

[0041] In formula (11), α and β are hyperparameters, indicating the weights of the two losses in the global contrast loss;

[0042] Step 2: Local Contrastive Learning

[0043] The ResNet-152 network and BERT model are used to extract local features of images and texts respectively, and the soft attention mechanism is used to learn the relevant information between image areas and text words. The soft attention mechanism calculates the correlation between each text word and each image area by weighted calculation, so there is no need to perform fine-grained annotation of text words and image areas.

[0044] In the scoring mechanism for flexibly selecting positive and negative samples, after the positive and negative samples are selected, if the distance between the positive sample pair and the anchor sample pair in the common subspace is reduced during both intra-modality and inter-modality contrastive learning, the distance between them may be too close, resulting in erroneous cross-modal semantic alignment. The distance between the anchor sample pair and the positive sample pair is reduced through intra-modality contrastive learning, and the distance between the anchor sample pair and the positive sample pair is expanded through inter-modality contrastive learning, thereby achieving the goal of reducing the distance between semantically similar data in the common subspace while preventing erroneous semantic alignment.

[0045] The specific steps of the step 2 local contrast learning are:

[0046] First, through the text word feature w i and image region feature r j The cosine similarity between the two is used to calculate the soft attention factor λ ij :

[0047]

[0048] In formula (12), R represents the total number of regions contained in the image, and the image region aligned with the i-th text word is Then the text t i All words and images in v i The similarity between all regions in is defined as:

[0049]

[0050] In formula (13), W represents the total number of words in the sentence, and the text-image local contrast loss is defined as:

[0051]

[0052] Similar to formulas (12) and (13), the soft attention factor λ between the image region feature and the text word feature is defined as ij , according to the soft attention factor λ ij Calculate image v i All areas and text in t i The similarity S between all words in vt (v,t), the image-text local contrast loss is defined as:

[0053]

[0054] For each image v in the current batch M i and text t i For contrastive learning, the local contrast loss is defined as:

[0055]

[0056] By minimizing the loss function in formula (16), the detailed semantic information between the image and the text can be fully retained, and local semantic alignment can be completed, thereby further reducing the distance between the image and text pairs in the common subspace. The total loss is defined as:

[0057] L=L Global +L Local (17)

[0058] In formula (17), L Global represents the global contrast loss, which consists of two items, where the global inter-modality contrast loss learns global semantic alignment, and the global intra-modality contrast loss reduces the distance between semantically similar data in the common subspace; L Local Represents local contrast loss, and uses soft attention mechanism to focus on the fine-grained information of images and texts to complete detailed semantic alignment;

[0059] The gradient descent method is used for training on the public dataset. The training is stopped when the total loss function L drops to the minimum, and the neural network parameters obtained by training are saved. When testing, the neural network parameters are loaded and the test dataset is input for testing. In the image retrieval text task, for each image, the text with the highest similarity to its cosine is selected as the retrieval result; in the text retrieval image task, for each text, the image with the highest similarity to its cosine is selected as the retrieval result.

[0060] An electronic device comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method as described above.

[0061] A computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method as described above.

[0062] The beneficial effect of the present invention is to propose a new scoring mechanism to solve the problem that the selection of positive and negative samples in the current contrastive learning method is not flexible enough, and to propose a cross-modal retrieval method based on global-local contrastive learning, which solves the problem that the existing method ignores the detailed semantic information between data. Based on the above method, the gradient descent method is used for training on a public data set, and the training is stopped when the loss function is reduced to the minimum and tends to be stable, and the relevant network parameters obtained by the training are saved. When testing, the relevant network parameters are loaded and the test data set is input for testing. In the image retrieval text task, each picture selects the text with the highest similarity as the retrieval result; in the text retrieval image task, each text selects the picture with the highest similarity as the retrieval result. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Schematic diagram of the scoring mechanism of the present invention. DETAILED DESCRIPTION

[0064] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0065] The present invention adopts the following methods and steps to realize cross-modal retrieval based on global-local contrastive learning:

[0066] 1. Global Contrastive Learning

[0067] Existing contrastive learning-based methods improve the discriminative performance of the model by learning a large number of samples. However, such methods are not flexible enough when selecting samples, resulting in semantically similar data being far apart in the common subspace. Unlike previous work, the present invention adopts a scoring mechanism to make one instance correspond to multiple positive samples, avoiding semantically similar data from being mapped to distant positions in the common subspace. On this basis, the global features of the image and text are input into the variational autoencoder, and the generated features are added to the positive sample set for learning, thereby further enhancing the discriminative ability of the model.

[0068] (1) Flexible scoring mechanism for selecting positive and negative samples

[0069] In order to improve the model's discriminative performance and avoid semantically similar data being far apart in the common subspace, this paper proposes a scoring mechanism for flexibly selecting positive and negative samples. The main idea is as follows: Figure 1 shown.

[0070] Figure 1 The main idea of ​​the scoring mechanism

[0071] in, Figure 1 (a) represents the initial distribution of multimodal data in the common subspace. In real scenarios, an image is described by multiple semantically similar texts, and one text describes multiple similar images; however, Figure 1 As shown in (b), the existing cross-modal retrieval method based on contrastive learning is not flexible enough in selecting samples. Many samples with similar semantics to the anchor data are regarded as negative samples for learning, resulting in semantically similar data (red and yellow samples in the figure) that should be in close positions in the common subspace being mapped to distant positions, resulting in poor discrimination ability of the model. To solve this problem, this paper adopts a scoring mechanism to flexibly select positive and negative samples. First, m sample pairs are randomly selected in the dataset to construct a batch M. Sample pairs s i and sample pairs s j The similarity score Sc between ij for:

[0072] Sc ij =min(cos(v i, v j ), cos(t i , t j )) (1)

[0073] In formula (1), cos(a, b) = (a·b) / (||a||2·||b||2) represents the cosine similarity between two samples. Only when the cosine similarities between the images and texts of both sample pairs are high, the similarity score Sc ij takes a larger value, and the sample pair s i includes the image v i and the text t i . After that, according to the similarity score Sc ij , positive and negative samples are selected for each sample pair s i . The positive sample set of s i is P i , and P i contains the top K sample pairs with the highest similarity scores to s i in the current batch; the negative sample set of s i is defined as N i , and N i is the complement of P i in the current batch;

[0074] P i = TopK(Sc ij ), j = 1, 2,..., m (2)

[0075]

[0076] In formula (3), is the complement symbol. Through the scoring mechanism, the top K sample pairs with higher similarity scores are selected as the positive samples of the anchor data, and the remaining sample pairs are used as the negative samples of the anchor data, thus realizing the flexible selection of positive and negative samples.

[0077] It should be noted that when the selection of positive and negative samples is completed, if the distances between the positive sample pairs and the anchor sample pairs in the common subspace are reduced both in intra-modal and inter-modal contrast learning, it may cause their distances to be too close, resulting in incorrect cross-modal semantic alignment (i.e., d1 < d2 in the figure), as Figure 1 (c) shows. To avoid this problem, in the subsequent method, the distance between the anchor sample pairs and the positive sample pairs is reduced through intra-modal contrast learning, and the distance between the anchor sample pairs and the positive sample pairs is enlarged through inter-modal contrast learning, so as to realize reducing the distance between semantically similar data in the common subspace while preventing incorrect semantic alignment, as Figure 1 (d) shows.

[0078] (2) Global inter-modal contrast learning

[0079] Global inter-modal contrastive learning aims to reduce the distance between paired images and texts in the common subspace and complete global semantic alignment. In order to avoid incorrect cross-modal semantic alignment caused by too close distances between sample pairs, the distance between two samples in the same sample pair is reduced during inter-modal contrastive learning, and the distance between two samples in different sample pairs is expanded; the normalized temperature scaled cross entropy loss is used as the loss function, the numerator of the loss function is the cosine similarity of the positive sample pairs, and the denominator of the loss function is the cosine similarity between the negative sample pairs. By minimizing the loss function, the cosine similarity between positive sample pairs can be increased, and the cosine similarity between negative samples can be reduced, thereby reducing the distance between positive samples and expanding the distance between negative samples in the common subspace. The image-text global inter-modal contrastive loss is defined as for:

[0080]

[0081] Similarly, the text-image global inter-modality contrast loss is defined as:

[0082]

[0083] In formulas (4) and (5), τ represents the temperature hyperparameter. For each image v in the current batch M, i and text t i For contrastive learning, the global inter-modal contrast loss is defined as:

[0084]

[0085] By minimizing the loss function in formula (6), the distance between image-text pairs in the common subspace can be reduced, thus achieving global semantic alignment.

[0086] (3) Global intra-modal contrastive learning

[0087] Global intra-modal contrastive learning aims to map semantically similar data to similar positions in a common subspace, thereby improving the model's discriminative ability.

[0088] First, the scoring mechanism is used to select the top K sample pairs with the highest similarity scores to construct a positive sample set. Then, in order to enable the anchor samples to learn positive samples with higher similarity during intra-modal contrast learning and further improve the model's discrimination ability, the initial image v is converted into i and text t i Input variational autoencoder to get the generated image and generate text And will generate the image and generate text Add to the positive sample set; new positive sample set P i ' is expressed as:

[0089]

[0090] In formula (7), is a pair of generated samples consisting of a generated image and a generated text. The global intra-modal contrastive learning uses the normalized temperature-scaled cross entropy loss as the loss function. The image-image global intra-modal contrastive loss is defined as:

[0091]

[0092] Similarly, the text-text global intra-modality contrastive loss is defined as:

[0093]

[0094] In formulas (8) and (9), K+1 is the number of positive samples selected by the scoring mechanism, and 1 means there is one positive sample generated by the variational autoencoder. For each image v in the current batch, i and text t i For contrastive learning, the global intra-modal contrastive loss is defined as:

[0095]

[0096] By minimizing the loss function in formula (10), the distance between the anchor data and the positive sample set in the feature subspace can be reduced, and the distance between the anchor data and the negative sample set in the feature subspace can be expanded, thereby improving the discriminative performance of the model. The global contrast loss is defined as:

[0097] L Global =αL GInter +βL GIntra (11)

[0098] In formula (11), α and β are hyperparameters, which represent the weights of the two losses in the global contrast loss.

[0099] 2. Local Contrastive Learning

[0100] In recent years, cross-modal retrieval methods based on the combination of coarse and fine granularity have gradually become a research hotspot. This type of method introduces the attention mechanism into cross-modal retrieval, explores the impact of local related information between different modal data on the model, and provides a new idea for improving retrieval accuracy. Inspired by this, in order to enhance the detailed semantic alignment between images and texts, local contrast learning is focused on on the basis of global contrast learning. Local contrast learning can further promote the semantic association between image and text pairs, thereby improving the retrieval accuracy of the model. To achieve this goal, the ResNet-152 network and the BERT model are used to extract local features of images and texts respectively, and the soft attention mechanism is used to learn the related information between image regions and text words. The soft attention mechanism calculates the correlation between each text word and each image region by weight, so there is no need to perform fine-grained annotations on text words and image regions. The specific method is:

[0101] First, through the text word feature w i and image region feature r j The cosine similarity between the two is used to calculate the soft attention factor λ ij :

[0102]

[0103] In formula (12), R represents the total number of regions contained in the image, and the image region aligned with the i-th text word is Then the text t i All words and images in v i The similarity between all regions in is defined as:

[0104]

[0105] In formula (13), W represents the total number of words in the sentence, and the text-image local contrast loss is defined as:

[0106]

[0107] Similar to formulas (12) and (13), the soft attention factor λ between the image region feature and the text word feature is defined as ij , according to the soft attention factor λ ij Calculate image v i All areas and text in t i The similarity S between all words in vt (v,t), the image-text local contrast loss is defined as:

[0108]

[0109] For each image v in the current batch M i and text ti For contrastive learning, the local contrast loss is defined as:

[0110]

[0111] By minimizing the loss function in formula (16), the detailed semantic information between the image and the text can be fully retained, and local semantic alignment can be completed, thereby further reducing the distance between the image and text pairs in the common subspace. The total loss is defined as:

[0112] L=L Global +L Local (17)

[0113] In formula (17), L Global represents the global contrast loss, which consists of two items, where the global inter-modality contrast loss learns global semantic alignment, and the global intra-modality contrast loss reduces the distance between semantically similar data in the common subspace; L Local Represents local contrast loss, and uses soft attention mechanism to focus on the fine-grained information of images and texts to complete detailed semantic alignment;

[0114] The gradient descent method is used for training on the public dataset. The training is stopped when the total loss function L drops to the minimum, and the neural network parameters obtained by training are saved. When testing, the neural network parameters are loaded and the test dataset is input for testing. In the image retrieval text task, for each image, the text with the highest similarity to its cosine is selected as the retrieval result; in the text retrieval image task, for each text, the image with the highest similarity to its cosine is selected as the retrieval result.

Claims

1. A cross-modal retrieval method based on global-local contrastive learning, characterized in that The steps include: Step 1: Global Contrastive Learning A scoring mechanism is used to make one instance correspond to multiple positive samples. The global features of the image and text are input into the variational autoencoder, and the generated features are added to the positive sample set for learning, thereby further enhancing the model's discriminative ability. (1) Flexible scoring mechanism for selecting positive and negative samples A scoring mechanism is used to flexibly select positive and negative samples. First, m sample pairs are randomly selected from the data set to construct a batch M. Sample pairs s i and sample pairs s j The similarity score Sc between ij for: Sc ij =min(cos(v i ,v j ),cos(t i ,t j )) (1) In formula (1), cos(a,b)=(a·b) / (||a||2·||b||2) represents the cosine similarity between two samples. The similarity score Sc is high only when the cosine similarity between the image and the text of the two sample pairs is high. ij Take the larger value, the sample pair s i Include image v i and text t i , then, according to the similarity score Sc ij For each sample pair s i Select positive and negative samples, s i The positive sample set is P i , P i Contains the current batch with s i The top K sample pairs with the highest similarity scores; definition s i The negative sample set is N i , N i P in the current batch i The complement of P i =TopK(Sc ij ),j=1,2,...,m (2) In formula (3), The symbol for the complement set is used. Through the scoring mechanism, K sample pairs with higher similarity scores are selected as positive samples of the anchor data, and the remaining sample pairs are used as negative samples of the anchor data, thereby realizing the flexible selection of positive and negative samples. (2) Global inter-modality contrastive learning Global inter-modal contrastive learning aims to reduce the distance between paired images and texts in the common subspace and complete global semantic alignment. In inter-modal contrastive learning, the distance between two samples in the same sample pair is reduced, and the distance between two samples in different sample pairs is expanded. Normalized temperature scaled cross entropy loss is used as the loss function. The numerator of the loss function is the cosine similarity of the positive sample pair, and the denominator of the loss function is the cosine similarity between the negative sample pairs. By minimizing the loss function, the cosine similarity between the positive sample pairs can be increased, and the cosine similarity between the negative samples can be reduced, thereby reducing the distance between the positive samples and expanding the distance between the negative samples in the common subspace. The image-text global inter-modal contrastive loss is defined as for: Similarly, the text-image global inter-modality contrast loss is defined as: In formulas (4) and (5), τ represents the temperature hyperparameter. For each image v in the current batch M, i and text t i For contrastive learning, the global inter-modal contrast loss is defined as: By minimizing the loss function in formula (6), the distance between image-text pairs in the common subspace can be reduced, thereby achieving global semantic alignment; (3) Global intra-modal contrastive learning Global intra-modal contrastive learning aims to map semantically similar data to similar positions in the common subspace, thereby improving the model's discriminative ability; First, the scoring mechanism is used to select the top K sample pairs with the highest similarity scores to construct a positive sample set. Then, in order to enable the anchor samples to learn positive samples with higher similarity during intra-modal contrast learning and further improve the model's discrimination ability, the initial image v is converted into i and text t i Input the variational self-encoder to get the generated image and generate text And will generate the image and generate text Add to the positive sample set; new positive sample set P i ′ is expressed as: In formula (7), is a pair of generated samples consisting of a generated image and a generated text. The global intra-modal contrastive learning uses the normalized temperature-scaled cross entropy loss as the loss function. The image-image global intra-modal contrastive loss is defined as: Similarly, the text-text global intra-modality contrastive loss is defined as: In formulas (8) and (9), K+1 is the number of positive samples selected by the scoring mechanism, and 1 means there is one positive sample generated by the variational autoencoder. For each image v in the current batch, i and text t i For contrastive learning, the global intra-modal contrastive loss is defined as: By minimizing the loss function in formula (10), the distance between the anchor data and the positive sample set in the feature subspace can be reduced, and the distance between the anchor data and the negative sample set in the feature subspace can be expanded, thereby improving the discriminative performance of the model. The global contrast loss is defined as: L Global =αL GInter +βL GIntra (11) In formula (11), α and β are hyperparameters, indicating the weights of the two losses in the global contrast loss; Step 2: Local contrastive learning The ResNet-152 network and BERT model are used to extract local features of images and texts respectively, and the soft attention mechanism is used to learn the relevant information between image areas and text words. The soft attention mechanism calculates the correlation between each text word and each image area by weighted calculation, so there is no need to perform fine-grained annotation of text words and image areas.

2. The cross-modal retrieval method based on global-local contrastive learning according to claim 1, characterized in that: In the scoring mechanism for flexibly selecting positive and negative samples, after the positive and negative samples are selected, if the distance between the positive sample pair and the anchor sample pair in the common subspace is reduced during both intra-modality and inter-modality contrastive learning, the distance between them may be too close, resulting in erroneous cross-modal semantic alignment. The distance between the anchor sample pair and the positive sample pair is reduced through intra-modality contrastive learning, and the distance between the anchor sample pair and the positive sample pair is expanded through inter-modality contrastive learning, thereby achieving the goal of reducing the distance between semantically similar data in the common subspace while preventing erroneous semantic alignment.

3. The cross-modal retrieval method based on global-local contrastive learning according to claim 1, characterized in that: The specific steps of the step 2 local contrast learning are: First, through the text word feature w i and image region feature r j The cosine similarity between the two is used to calculate the soft attention factor λ ij : In formula (12), R represents the total number of regions contained in the image, and the image region aligned with the i-th text word is Then the text t i All words and images in v i The similarity between all regions in is defined as: In formula (13), W represents the total number of words in the sentence, and the text-image local contrast loss is defined as: Similar to formulas (12) and (13), the soft attention factor λ between the image region feature and the text word feature is defined as ij , according to the soft attention factor λ ij Calculate image v i All areas and text in t i The similarity S between all words in vt (v,t), the image-text local contrast loss is defined as: For each image v in the current batch M i and text t i For contrastive learning, the local contrast loss is defined as: By minimizing the loss function in formula (16), the detailed semantic information between the image and the text can be fully retained, and local semantic alignment can be completed, thereby further reducing the distance between the image and text pairs in the common subspace. The total loss is defined as: L=L Global +L Local (17) In formula (17), L Global represents the global contrast loss, which consists of two items, where the global inter-modality contrast loss learns global semantic alignment, and the global intra-modality contrast loss reduces the distance between semantically similar data in the common subspace; L Local Represents local contrast loss, and uses soft attention mechanism to focus on the fine-grained information of images and texts to complete detailed semantic alignment; The gradient descent method is used for training on the public data set. The training is stopped when the total loss function L is minimized, and the neural network parameters obtained by training are saved. When testing, the neural network parameters are loaded and the test data set is input for testing. In the image retrieval text task, the text with the highest similarity to the cosine of each image is selected as the retrieval result. In the text retrieval image task, for each text, the image with the highest cosine similarity is selected as the retrieval result.

4. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 3.

Images