Aggregated neighborhood embedded image text matching method based on clustering prototype guidance

Abstract

The invention discloses an aggregation neighborhood embedding image text matching method based on clustering prototype guidance, relates to the technical field of computer vision and natural language processing, and provides a full-granularity prototype guidance neighborhood embedding framework. The framework dynamically learns a unified representation space through clustering prototype momentum contrast learning, and relieves a soft positive sample problem; global and local feature joint extraction and an adaptive aggregation network are introduced, so that the global information understanding capability is improved; an intra-modal correlation degree matrix and an inter-modal correlation degree matrix are constructed, and the neighborhood relation of sample instances is deeply mined; and by adopting a momentum contrast learning technology, the negative sample capacity is expanded, and the feature discrimination capability is improved. Experimental results show that the performance of the FPNE on Flickr30K and MS-COCO data sets is superior to that of an existing most advanced method, the image-text matching precision and efficiency are remarkably improved, and high robustness and wide applicability are shown.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and natural language processing, and in particular to an aggregated neighborhood embedding image-text matching method based on clustering prototype guidance. Background Art

[0002] The image-text matching task aims to bridge the semantic gap between computer vision and natural language processing. It is committed to achieving accurate semantic alignment between images and texts, laying the foundation for many applications of multimodal learning, including cross-modal retrieval, image description generation, text-guided image synthesis, and visual question answering. The core challenge lies in how to accurately and efficiently learn the complex semantic correspondence between images and texts and achieve high-quality cross-modal semantic mapping.

[0003] Existing image-text matching methods are mainly divided into two paradigms: independent embedding and interactive embedding. The independent embedding method encodes the complete image and text separately into a shared embedding space, and quantifies the semantic similarity between modalities by comparing global embeddings. This method can pre-compute representations and cache indexes. Due to its high efficiency and considerable accuracy, it has been favored in large-scale retrieval applications in recent years.

[0004] Interactive embedding methods focus on learning fine-grained alignments between salient visual elements in images and text snippets, and inferring overall image-text similarity. This method can capture more subtle and rich semantic associations, but has high computational complexity and may face the challenge of significantly increased inference time.

[0005] However, existing methods often ignore the mining of high-order associations and subtle differences between semantically similar instances, which is very helpful for image-text matching tasks:

[0006] (a) Semantically similar instances within a batch: Although these instances differ in specific details (such as baseball and tennis), they are highly related in the core semantics of "preparing to hit the ball" and contain rich shared knowledge. This highlights the necessity of modeling high-order semantic relationships between instances;

[0007] (b) Soft positive sample problem: two image-text pairs depicting similar baseball scenes are shown, but they are incorrectly labeled as negative samples despite their semantic closeness. The existence of such "soft positive samples" may interfere with the learning process of the model and affect its ability to recognize subtle semantic differences;

[0008] (c) Soft negative sample problem: Shows one of the top 5 results retrieved using the interactive embedding method (CHAN). Although the text and image match well in local details, there are differences in overall semantics, forming "soft negative samples" that may cause the model to confuse global and local semantic understanding.

[0009] Existing methods are inadequate in handling high-order associations and subtle differences between semantically similar instances, especially when dealing with soft positive samples (semantically similar samples that are incorrectly labeled) and soft negative samples (samples that are retrieved with partial matches but inconsistent overall semantics). In addition, how to fully model the neighborhood relationship between semantically similar instances in the training batch so that the model can learn high-order shared knowledge is still an urgent problem to be solved. Summary of the invention

[0010] The purpose of the present invention is to provide an image-text matching method based on clustered prototype-guided aggregated neighborhood embedding. Specifically, we propose a full-granularity prototype-guided neighborhood embedding framework (FPNE). This framework dynamically learns a unified representation space through clustered prototype momentum contrastive learning to alleviate the problem of soft positive samples; introduces global and local feature joint extraction and adaptive aggregation network to improve the ability to understand global information; constructs intra-modal and inter-modal correlation matrices to deeply explore the neighborhood relationship of sample instances; and uses momentum contrastive learning technology to expand the capacity of negative samples and improve feature discrimination ability. Experimental results show that FPNE outperforms the existing most advanced methods on Flickr30K and MS-COCO datasets, significantly improves the accuracy and efficiency of image-text matching, and shows strong robustness and wide applicability.

[0011] In order to solve the above technical problems, the present invention provides an aggregated neighborhood embedding image text matching method based on cluster prototype guidance, comprising the following steps:

[0012] S1. Construct an aggregated full-granularity semantic representation learning network, including joint extraction of global and local features, adaptive local feature aggregation guided by global semantics, and full-granularity feature gating fusion;

[0013] S2, using cluster prototype momentum contrast learning network, through online clustering, dynamic learning of unified cluster prototypes in the joint representation space of images and texts;

[0014] S3, image and text features are projected into the shared prototype space, and the similarity with each prototype is calculated;

[0015] S4. Use the Sinkhorn algorithm to calculate the soft assignment probability distribution and use the cross entropy loss as the comparative learning target;

[0016] S5, introduces two momentum encoders to generate a large dynamic repository as additional negative samples;

[0017] S6. Instance neighborhood relationship modeling uses global and local features to construct intra-modal and inter-modal correlation matrices to characterize the semantic similarity between instances.

[0018] Furthermore, in said S1:

[0019] The specific steps of joint extraction of global and local features are as follows: for the input image I, the bottom-up attention model [1] is used to extract the first K = 36 most significant regional features using the Faster R-CNN network pre-trained on the Visual Genomes dataset. Each regional feature is mapped to a d-dimensional joint semantic space through a fully connected linear projection layer to obtain the image region representation. Where n r is the number of regional features;

[0020] For the input sentence T, the pre-trained BERT model is used as the text encoder to extract word-level feature representations, and then a fully connected linear projection layer is used to map these word features into the same d-dimensional joint semantic space as the image, resulting in Where n t is the number of word-level features;

[0021] Fine-grained semantic association feature enhancement is performed for visual regions and text words. Taking the image modality as an example, a semantic relationship graph between visual regions in an image is constructed, and a graph attention network is proposed to learn the contextual semantics of regional features. The nodes of the graph are sets of regional features, and the edges represent the semantic associations between regions.

[0022] Original image region representation

[0023] After the graph attention network, we obtain fine-grained semantic association enhanced regional features

[0024] Similarly, we can obtain fine-grained semantically related enhanced word features

[0025] The pre-trained CLIP model is introduced to extract the global feature representation, and the CLIP (ViT / B-32) model is directly used to extract the visual semantic embedding without fine-tuning; through the fully connected linear projection layer, we get the global feature representation v of the image and text g ∈R d and t g ∈R d , together with the regional features and word features, are located in the d-dimensional joint semantic space.

[0026] Furthermore, in said S1:

[0027] The adaptive local feature aggregation guided by global semantics is as follows:

[0028] The input image is passed through the feature extraction network to obtain the local feature vector as the query, and the image is divided into several salient regions. The feature vector of each region is regarded as a codeword to form the codebook of the image.

[0029] Perform multiple linear transformations on the query and codebook, and calculate the similarity between the transformed query and codeword to obtain multiple sets of soft-assigned coding coefficient vectors;

[0030] The vector is then mean-pooled to obtain the final coding coefficient vector;

[0031] Finally, through softmax normalization, this vector is converted into a set of fine-grained adaptive weights, which are used to perform weighted pooling on the features of the salient regions to obtain the encoded representation of the image.

[0032] Furthermore, in said S1:

[0033] Capture subtle semantic differences between samples through cluster mapping, and then compare them in the cluster semantic space;

[0034] Define a set of trainable prototype vectors P=p1,p2,…,p in the joint representation space of image and text k , where k is the number of prototypes; these prototype vectors can be regarded as low-dimensional projections of the entire dataset, storing frequent, common, and transferable features in the dataset; given a pair of image embeddings V = v1,…,v m and text embedding T = t1,…,t m , project it into the unified prototype space, and get the similarity scores of the image and text with each prototype:

[0035]

[0036]

[0037] in Each row represents the similarity score between a sample and k prototypes;

[0038] Use the Sinkhorn algorithm to get the soft assignment matrix of image and text from the similarity scores Each sample is softly assigned to multiple prototypes, and the sum of the assignment probabilities is 1; the Sinkhorn algorithm assigns u v and u t Convert to soft allocation matrix;

[0039] Compute the prototype-guided cross-modal alignment loss (PGA) of image perspective and text perspective;

[0040] For each image view, the soft assignment matrix D of the text embedding t As the "positive sample" label, the similarity score u between the image and the prototype vAs the prediction value, negative sampling comparison is performed; the goal is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs; minimize the negative cross entropy loss of the two, that is, maximize the similarity of positive sample pairs, and naturally minimize the similarity of negative sample pairs. The contrast loss of image perspective can be calculated by the following formula:

[0041]

[0042] Where m is the batch size, k is the number of prototypes, τ is the temperature coefficient, represents the probability that the i-th text embedding is assigned to the j-th prototype, represents the similarity score between the i-th image embedding and the j-th prototype;

[0043] Similarly, for the text perspective, the soft assignment matrix D of the image embedding is v As the "positive sample" label, the similarity score u between the text and the prototype t As the predicted value, negative sampling comparison is performed, and the contrast loss of the text perspective can be calculated by the following formula:

[0044]

[0045] The final prototype-guided cross-modal alignment loss is the sum of the image perspective and text perspective losses:

[0046]

[0047] Furthermore, it also includes building two dynamic repositories to decouple the repository size from the mini-batch size, so the size of negative samples can be much larger than the mini-batch size;

[0048] Let θ v With θ t They are respectively the visual encoder f v and the text encoder f t The parameters of the momentum encoder are introduced for the visual and language modalities respectively. (Parameters are )and (Parameters are ), whose parameters are updated based on momentum:

[0049]

[0050] represents the momentum coefficient, which takes a relatively large value and is set in this paper as

[0051] Create two dynamic repositories and Used to store features extracted by the visual momentum encoder And the characteristics of the text momentum encoder N v and N t Represent the size of the visual and text dynamic repositories respectively. In each round of training, the data of the current batch is input into the momentum encoder to obtain the momentum feature representation, and then input into the dynamic repository to update the repository, and the oldest batch of data is removed from the repository. This operation is performed after calculating the loss;

[0052] Using InfoNCE loss as the objective function, given each image in the data batch, its paired text is regarded as a positive sample, and B t All stored samples are regarded as negative samples, and we define the momentum contrast loss (MCL) for image to text as:

[0053]

[0054] Among them, v i Represents the feature vector of the current batch of images, Represents the momentum feature vector of its paired text. The similarity function sim(·,·) uses the dot product of two vectors to calculate their similarity score;

[0055] Similarly, given each text in a batch of data, we define the momentum contrastive loss for text to image as:

[0056]

[0057] Among them, t i Represents the feature vector of the current batch of text, represents the momentum feature vector of its paired image;

[0058] The final momentum contrast loss is:

[0059]

[0060] Furthermore, in S6, each instance and its related neighbors are constructed as a neighborhood subgraph, and a graph neural network is introduced to perform modal interactive learning to fully explore the neighborhood relationship of the sample instance;

[0061] Incorporate the semantic knowledge shared by samples into cross-modal feature interactions, construct a multimodal instance neighborhood semantic association graph, and apply graph neural networks on the graph to perform semantic interactions within and between modalities.

[0062] The instance neighborhood semantic association graphs are constructed within and between modalities respectively. The intra-modal association graph uses the Euclidean distance of the global visual / text embedding to quantify the semantic similarity between instances:

[0063]

[0064] in denote the global feature vectors of the i-th and j-th images respectively, denote the global feature vectors of the i-th and j-th texts, respectively, |·|2 denotes the l2 norm, and ε>0 is the temperature parameter;

[0065] The fine matching of local features is used to measure the semantic association of cross-modal instance neighborhoods. Specifically, given a pair of image-text and its region-word similarity matrix, the text word (or visual region) with the highest matching degree is found for each region (or each word), and then these maximum matching scores are averaged as the measure of the semantic association of cross-modal instance neighborhoods:

[0066]

[0067] where n r ,n t Represent the number of image regions and the number of text words, respectively. are the τth region feature of the ith image and the ηth word feature of the jth text respectively;

[0068] The final multimodal instance neighborhood semantic association graph can be expressed as:

[0069]

[0070] Where m is the batch size;

[0071] After constructing the multimodal instance neighborhood semantic association graph, a multimodal graph neural network is applied to perform instance neighborhood semantic interaction on the graph.

[0072] The graph convolutional network (GCN) is introduced to realize the intra-modal neighborhood semantic interaction of image modality and text modality. For the image modality, the l-th layer of GCN can be expressed as:

[0073]

[0074] in represents the image node feature matrix of the lth layer, Represents the neighborhood semantic association graph of image modality instances, is the learnable weight matrix of layer l, σ(·) is the nonlinear activation function,

[0075] are the diagonal elements of the degree matrix;

[0076] Similarly, for the text modality, the l-th layer GCN can be expressed as:

[0077]

[0078] Use the joint modality GCN to aggregate neighborhood structure knowledge, concatenate the outputs of the image modality GCN and the text modality GCN in the batch dimension, and then input the concatenated features into the joint modality GCN for cross-modal interaction, embedding instance neighborhood correlation into the feature learning process:

[0079]

[0080] in is the joint modality node feature matrix of the lth layer, is a multimodal instance neighborhood semantic association graph, is a learnable weight matrix;

[0081] A joint-modality graph attention network (GAT) is applied to adaptively and finely capture neighborhood semantic associations, and the output of the joint-modality GCN is used as the input of the joint-modality GAT.

[0082] Optimize the original multimodal instance neighborhood semantic association graph S to obtain the optimized adjacency matrix Sort the element values ​​in S, for each submatrix S I→I , S I→T , S T→I and S T→T , set the elements with values ​​in the last 50% to -∞, and multiply the elements with values ​​in the first 50% by a magnification factor α>1, and get the optimized adjacency matrix S:

[0083]

[0084] Among them, Λ(·) represents the operation of taking the median;

[0085] After obtaining the optimized adjacency matrix After that, it replaces the original S for instance neighborhood semantic interaction. Incorporated into GAT's self-attention score calculation:

[0086]

[0087] in is the query, key, and value matrix obtained by linearly transforming the output of the joint modality GCN, d k is the dimension of the key vector;

[0088] After instance neighborhood semantic interaction, the final enhanced visual-language embedding is obtained, denoted as and Where m is the batch size;

[0089] Adopt hard negative triplet loss Ltri , ensuring that the distance between matching image-text pairs is closer than the distance between the most difficult image-text pair to match:

[0090] L tri (v,t)=[γ-s(v,t)+s(v,t - )] + +[γ-s(v,t)+s(v - ,t)] + ,

[0091] where γ represents the marginal hyperparameter, s(·,·) is used to calculate the cosine similarity of visual-language embeddings, and [·] + ≡max(·,0), (v,t) is a positive image-text pair, v - = argmax x≠v s(x,t) and t - = argmax y≠t s(v,y) are the hardest negative image and negative text in the batch, respectively, used to construct the negative image-text pair (v - ,t) and (v,t - );

[0092] Not only the triplet loss is calculated for the original visual-language embedding, but also the loss is calculated for the enhanced embedding and cross-aligned. The total instance neighborhood semantic interaction loss is defined as follows:

[0093]

[0094] in, Represents the prototype-guided cross-modal alignment loss for the enhanced embedding calculation, and the final objective function is:

[0095] L=L PGA +L MCL +L NSI ,.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] 1. Innovatively apply the cluster prototype contrastive learning method to the image-text matching task. By unifying the prototype to better align visual and text embeddings, the "soft positive sample" problem is alleviated. This method not only improves the matching accuracy, but also enhances the model's robustness to semantically similar samples.

[0098] 2. We propose an aggregated full-grained semantic representation learning framework, which integrates the advantages of coarse-grained and fine-grained retrieval methods and achieves the effective capture and utilization of multi-grained semantic information.

[0099] 3. We designed and implemented an instance neighborhood relationship modeling method. By deeply mining the high-order relationships between instances, we enabled the model to learn richer shared knowledge and thus obtain a better overall embedding representation.

[0100] 4. Introduce momentum contrastive learning (MCL) technology into image text representation learning. By expanding the capacity of negative samples, we effectively improve the discriminative ability of features.

[0101] 5. Extensive experiments on two widely used public benchmarks for image-text matching, namely Flickr30K and MS-COCO, show that our proposed framework achieves SOTA performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 Schematic diagram of semantically similar instances within a batch, soft positive sample problem, and soft negative sample problem;

[0103] Figure 2 It is a schematic diagram of the overall framework of the present invention;

[0104] Figure 3 Schematic diagram of the global semantics-guided adaptive regional feature pooling method;

[0105] Figure 4 is the parameter analysis of k in Equation 8 on two data sets;

[0106] Figure 5 is the parameter analysis of α in Equation 21 on two data sets;

[0107] Figure 6 A diagram showing the comparison of accuracy and speed obtained after executing all methods on the Flickr30K test set using the RTX3090 GPU. DETAILED DESCRIPTION

[0108] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0109] 1 Introduction

[0110] Instructions attached Figure 1Explain our motivation. Mining high-order associations and subtle differences between semantically similar instances is very helpful for image-text matching tasks: (a) Semantically similar instances within a batch: Although these instances differ in specific details (such as baseball and tennis), they are highly related in the core semantics of "preparing to hit the ball" and contain rich shared knowledge. This highlights the necessity of modeling high-order semantic relationships between instances. (b) Soft positive sample problem: Two image-text pairs describing similar baseball scenes are shown, which are incorrectly labeled as negative samples despite their semantic closeness. The existence of such "soft positive samples" may interfere with the learning process of the model and affect its ability to identify subtle semantic differences. (c) Soft negative sample problem: One of the top 5 results retrieved using the interactive embedding method (CHAN) is shown. Although the text and image match well in local details, there are differences in overall semantics, forming "soft negative samples", which may cause the model to confuse between global and local semantic understanding.

[0111] Image-text matching is a key task to bridge the semantic gap between computer vision and natural language processing. It is committed to achieving accurate semantic alignment between images and texts, laying the foundation for many applications of multimodal learning, including cross-modal retrieval, image description generation, text-guided image synthesis, and visual question answering. The core challenge is how to accurately and efficiently learn the complex semantic correspondence between images and texts and achieve high-quality cross-modal semantic mapping.

[0112] Existing image-text matching methods can be divided into two major paradigms: independent embedding and interactive embedding. Independent embedding methods encode the complete image and text into a shared embedding space respectively, and quantify the semantic similarity between modalities by comparing global embeddings. This method can pre-compute representations and cache indexes. Due to its high efficiency and considerable accuracy, it has been favored in large-scale retrieval applications in recent years. Interactive embedding methods focus on learning fine-grained alignment between salient visual elements in images and text fragments, and inferring overall image-text similarity. This method can capture more subtle and rich semantic associations, but it has high computational complexity and may face the challenge of significantly increased inference time.

[0113] However, existing methods often ignore the exploration of high-order associations and subtle differences between semantically similar instances. The image-text matching task can essentially be viewed as a contrastive learning problem, which aims to bring together the embedding representations of semantically related positive image-text pairs, while pushing away irrelevant negative pairs. Existing benchmark datasets (such as MS-COCO, Flickr30K, LAION-400M) have limitations in their construction, simply labeling paired samples as positive samples and all other samples as negative samples. In fact, there may be samples in the negative samples that are semantically related to the anchor point, such as Figure 1As shown in part (b) of . These samples are semantically very close to the positive samples paired with the anchor points, but are mistakenly labeled as negative samples. We call them "soft positive samples". The widespread "soft positive samples" in the dataset may cause trouble for model learning.

[0114] In addition, traditional interactive embedding methods often have some instances in reasoning that align well with the local fragments of the retrieval anchor but do not match the global semantics. Figure 1 As shown in part (c), the open source SOTA method CHAN with interactive embedding is used for image-to-text retrieval. The text in the figure is one of the top 5 texts retrieved by CHAN. The text is well aligned with the retrieved image in local segments, but there are differences in global semantics. This retrieved negative sample is very close to the anchor point and its positive sample in fine-grained semantics, which can easily confuse the model. We call it a "soft negative sample".

[0115] On the other hand, there may be a large number of semantically similar instances in the model training batch, such as Figure 1 As shown in part (a) of the figure, how to fully model the neighborhood relationship between instances so that the model can learn high-order shared knowledge is still an under-researched issue.

[0116] To this end, we propose a Full-grained Prototype-guided Neighborhood Embedding (FPNE) framework. The framework consists of the following key components:

[0117] First, we propose a cluster prototype momentum contrastive learning method. Through online clustering, unified cluster prototypes in the joint representation space of images and texts are dynamically learned. These unified prototypes serve as a bridge to better align visual and text embeddings. At the same time, this method avoids the problem that random negative sampling may mistake semantically similar sample pairs as negative samples, thereby reducing the possibility of the model learning wrong knowledge and alleviating the trouble caused by "soft positive samples" to a certain extent.

[0118] Secondly, to solve the problem that traditional interactive embedding methods lack global information understanding and generate "soft negative samples" in actual reasoning, we proposed an aggregated full-granularity semantic representation learning network. This network introduces the offline CLIP model to generate global representation, improves the model's global information understanding ability, and uses the global representation to guide local features for adaptive aggregation. Finally, the global representation and local representation are fused to obtain a full-granularity semantic embedding.

[0119] Furthermore, in order to fully model the neighborhood relationship of instances and enable the model to learn high-order shared knowledge, we modeled the neighborhood relationship of instances. Specifically, we constructed the intra-modal correlation matrix and the inter-modal correlation matrix between instances, and used these two matrices to fully explore and model the neighborhood relationship of sample instances. We constructed each instance and its intra-modal neighbors and inter-modal related instances into a neighborhood subgraph, and introduced a graph neural network for modal interaction learning.

[0120] In addition, inspired by MOCO, in order to expand the capacity of negative samples and improve the effect of contrastive learning, we introduced two momentum encoders, which are used to encode images and texts respectively, to produce large and consistent dynamic memory banks. These dynamic memory banks store a large number of image and text feature representations as negative samples for contrastive learning, which improves the discriminative ability of features.

[0121] By integrating these modules, our Full-grained Prototype-guided Neighborhood Embedding (FPNE) framework can fully exploit high-order associations and subtle differences between semantically similar instances at different stages, effectively improving the performance of image-text retrieval while maintaining high efficiency.

[0122] Our main contributions can be summarized as follows:

[0123] Innovatively apply cluster prototype contrastive learning method to image-text matching task. By unifying prototypes to better align visual and text embeddings, the "soft positive sample" problem is alleviated. This method not only improves matching accuracy, but also enhances the robustness of the model to semantically similar samples.

[0124] A framework for learning aggregated full-grained semantic representations is proposed, which integrates the advantages of coarse-grained and fine-grained retrieval methods and achieves effective capture and utilization of multi-grained semantic information.

[0125] We designed and implemented an instance neighborhood relationship modeling method. By deeply mining the high-order relationships between instances, we enabled the model to learn richer shared knowledge and thus obtain a better overall embedding representation.

[0126] We introduce momentum contrastive learning (MCL) technology into image text representation learning. By expanding the capacity of negative samples, we effectively improve the discriminative ability of features.

[0127] Extensive experiments on two widely used public benchmarks for image-text matching, namely Flickr30K and MS-COCO, demonstrate that our proposed framework achieves SOTA performance.

[0128] 2 Proposed method

[0129] In this section, I will introduce the proposed method in detail. The overall framework is as follows Figure 2 shown.

[0130] Figure 2 The proposed framework is a full-granularity prototype-guided neighborhood embedding framework, which includes the following key modules: First, the full-granularity semantic representation learning network is aggregated, including the joint extraction of global and local features, adaptive local feature aggregation guided by global semantics, and full-granular feature gating fusion. Second, the clustering prototype momentum contrastive learning network dynamically learns unified clustering prototypes in the image-text joint representation space through online clustering. These prototypes serve as a bridge to better align visual and text embeddings. Image and text features are projected into the shared prototype space, and the similarity with each prototype is calculated. The soft assignment probability distribution is calculated using the Sinkhorn algorithm, and the cross entropy loss is used as the contrastive learning target to enhance the feature representation ability. To expand the capacity of negative samples, two momentum encoders are introduced to generate a large dynamic repository as additional negative samples. Finally, the instance neighborhood relationship is modeled, and the intra-modal and inter-modal correlation matrices are constructed using global and local features to characterize the semantic similarity between instances. We construct each instance and its related neighbors as a neighborhood subgraph, and introduce a graph neural network for modal interaction learning to fully explore the neighborhood relationship of sample instances.

[0131] 2.1 Learning Full-Grained Semantic Representations

[0132] Existing image-text matching methods based on local feature extraction ignore the hierarchical aggregation of multi-level features, and simply fuse local features as equally important, lacking the guidance of global semantics. Methods based on global feature matching often lack the ability to perceive fine-grained local features. To solve this problem, we propose a learning framework that integrates full-grained semantics.

[0133] Under the guidance of global semantics, this framework can aggregate local features and effectively mine and fuse multi-level semantic information of images and texts. At the same time, through the mutual feedback of global and local features, the global semantic representation can obtain fine-grained semantic detail support, and the local feature representation can also fully absorb the guidance of global semantics to achieve effective flow and complementarity of cross-level semantics. 2.1.1 Joint extraction of global and local features

[0134] like Figure 3As shown in Figure 1, for each input image I, we use the bottom-up attention model and the Faster R-CNN network pre-trained on the Visual Genomes dataset to extract the top K = 36 most significant regional features. Through the fully connected linear projection layer, each regional feature is mapped to the d-dimensional joint semantic space to obtain the image region representation. Where n r is the number of regional features.

[0135] For each input sentence T, we use the pre-trained BERT model as the text encoder to extract word-level feature representations. Then, we use a fully connected linear projection layer to map these word features into the same d-dimensional joint semantic space as the image, and obtain Where n t is the number of word-level features.

[0136] We perform fine-grained semantic association feature enhancement for visual regions and text words respectively. Taking the image modality as an example, we construct a semantic relationship graph between visual regions in an image and propose a graph attention network to learn the contextual semantics of regional features. The nodes of the graph are sets of regional features, and the edges represent the semantic associations between regions. Original image region representation After the graph attention network, we obtain fine-grained semantic association enhanced regional features Similarly, we can obtain fine-grained semantically related enhanced word features

[0137] In order to fully obtain the global semantic information of images and texts, we introduce the pre-trained CLIP model to extract global feature representation. Considering the limitation of computing resources and the purpose of more fully verifying the effectiveness of this method, we directly use the CLIP (ViT / B-32) model to extract visual semantic embedding without fine-tuning. Specifically, through the fully connected linear projection layer, we get the global feature representation v of the image and text g ∈R d and t g ∈R d , together with the regional features and word features, are located in the d-dimensional joint semantic space.

[0138] 2.1.2 Adaptive Local Feature Aggregation Guided by Global Semantics

[0139] Figure 3Description of the method for adaptive regional feature pooling guided by global semantics First, the input image is passed through a feature extraction network (such as CLIP) to obtain a global feature vector as a query. At the same time, we divide the image into several salient regions, and the feature vector of each region is regarded as a codeword to form the codebook of the image. Next, we perform multiple linear transformations on the query and codebook, and calculate the similarity between the transformed query and codeword to obtain multiple sets of soft-assigned coding coefficient vectors. These vectors are then mean-pooled to obtain the final coding coefficient vector. Finally, through softmax normalization, we can convert this vector into a set of fine-grained adaptive weights, which are used to weighted pool the salient region features to obtain the encoded representation of the image.

[0140] We propose a global semantics-guided adaptive local feature pooling method that exploits a soft assignment encoding framework to learn fine-grained feature pooling weights. Figure 3 Taking image coding as an example, we represent the salient area of ​​the image as These regions are used as the codebook of the image. Each salient region feature vector r i (i∈[1,n r ]) is considered a codeword.

[0141] We use the image global feature vector v extracted by CLIP g ∈R d As a query, suppose is the soft-assigned coding coefficient vector of r with respect to codebook R.

[0142] Among them, the jth coding coefficient u j Represents the global feature vector v of the image g (query) in the jth salient region feature r j The degree of membership on is calculated as follows:

[0143]

[0144] In order to capture more potential patterns, we perform linear transformation on the query vector and codebook to obtain the query set and codebook set where m is the number of transforms. Then, for the kth transform, we calculate its soft-assigned coding coefficient vector The calculation is as follows:

[0145]

[0146] in is the codebook corresponding to the kth transformation The jth codeword in Represents query The degree of membership on this codeword.

[0147] Finally, we transform all the soft-assigned coding coefficient vectors Perform mean pooling to obtain the final soft allocation coding coefficient vector

[0148]

[0149] Among them, a v Indicates query v g Regarding the soft-assigned encoding coefficient vectors of codebook R, in order to convert these coefficient vectors into fine-grained feature pooling weights, we v Perform softmax processing:

[0150] w v =softmax(a v ) (4)

[0151] in, represents the learned image fine-grained feature pooling weights. Similarly, we can get the text fine-grained feature pooling weights

[0152] 2.1.3 Full-granularity feature gating fusion

[0153] In order to obtain the full-grained aggregate feature, we need to convert the fine-grained aggregate feature v f and t f With the global feature representation v g and t g A straightforward way to combine them is to weight them using scaling hyperparameters. However, to avoid manually tuning these hyperparameters, we let the model automatically learn the weighting coefficients through trainable weights. The formula for calculating the image full-granularity aggregate feature v is as follows:

[0154]

[0155] Where g is a gate value used to adaptively balance v f and v g The importance of [·,·] indicates the concatenation operation.

[0156] Similarly, the calculation formula of the full-granularity aggregate feature t of the text is:

[0157]

[0158] In this way, we can adaptively fuse fine-grained and global features to obtain full-grained aggregated feature representations of images and texts.

[0159] 2.2 Clustering Prototypes to Guide Cross-modal Alignment

[0160] In the past, mainstream methods in image-text matching tasks adopted a self-supervised contrastive learning framework and widely used objective functions such as InfoNCE loss

[26] and Hard Negative Triplet Loss [8]. However, these methods may mistake some sample pairs with similar actual semantics as negative samples during random negative sampling, causing the model to learn incorrect knowledge. Especially for Hard Negative Triplet Loss, the selected hard negative examples may be semantically close to the positive samples, which may cause learning confusion in the model. Figure 1 As shown in (c), although the two pairs are positive and negative samples, they are very similar in semantics. The obtained embedding vectors are difficult to distinguish and can be completely regarded as positive sample pairs. If they are mistakenly regarded as negative samples, it will interfere with model learning.

[0161] To solve this problem, we innovatively designed a cluster prototype-guided cross-modal alignment module. Unlike traditional contrastive learning that directly maximizes the similarity of positive samples, we first capture the subtle semantic differences between samples through cluster mapping, and then compare them in the cluster semantic space, so as to better utilize the intrinsic semantic structure information of the data.

[0162] Specifically, first, we define a set of trainable prototype vectors P = p1, p2, ..., p in the joint representation space of image and text k , where k is the number of prototypes. These prototype vectors can be regarded as low-dimensional projections of the entire dataset, storing frequent, common, and transferable features in the dataset. Given a pair of image embeddings V = v1,…,v m and text embedding T = t1,…,t m , we project it into the unified prototype space and get the similarity scores of the image and text with each prototype:

[0163]

[0164] in Each row represents the similarity score between a sample and k prototypes. Next, we use the Sinkhorn algorithm to obtain the soft assignment matrix of image and text from the similarity scores. Each sample is softly assigned to multiple prototypes, and the sum of the assignment probabilities is 1. The Sinkhorn algorithm can v and u tConverted into a soft assignment matrix, this soft assignment form can better capture the subtle semantic differences between data than hard assignment.

[0165] We then compute the prototype-guided cross-modal alignment loss (PGA) between image perspective and text perspective:

[0166] For the image view, we embed the soft assignment matrix D t As the "positive sample" label, the similarity score u between the image and the prototype v As the prediction value, negative sampling comparison is performed. The goal is to maximize the similarity of the positive sample pair and minimize the similarity of the negative sample pair. Specifically, we minimize the negative cross entropy loss of the two, that is, maximize the similarity of the positive sample pair (the prototype with a large text soft assignment probability should also have a large image similarity score), and naturally minimize the similarity of the negative sample pair (the prototype with a small text soft assignment probability should also have a small image similarity score). The contrast loss of the image perspective can be calculated by the following formula:

[0167]

[0168] Where m is the batch size, k is the number of prototypes, τ is the temperature coefficient, represents the probability that the i-th text embedding is assigned to the j-th prototype, represents the similarity score between the i-th image embedding and the j-th prototype.

[0169] Similarly, for the text perspective, we embed the soft assignment matrix D of the image v As the "positive sample" label, the similarity score u between the text and the prototype t As the predicted value, negative sampling comparison is performed. The contrast loss of text perspective can be calculated by the following formula:

[0170]

[0171] The final prototype-guided cross-modal alignment loss is the sum of the image perspective and text perspective losses:

[0172]

[0173] Through this clustering comparison method, our model can better utilize the intrinsic semantic structure of the data and learn more accurate cross-modal alignment knowledge.

[0174] 2.3 Dynamic Repository Assisted Image-Text Comparative Learning

[0175] Most current image-text matching methods rely only on samples from each mini-batch during training, which limits the size of negative samples and thus inhibits the representation capability of the model. Inspired by MoCo and related downstream work, we propose to build two dynamic repositories (one for each modality) to decouple the repository size from the mini-batch size, so that the size of negative samples can be much larger than the mini-batch size. In this way, we can save more negative sample embeddings to assist image-text contrast learning.

[0176] Specifically, let θ v With θ t They are the visual encoder f v and the text encoder f t In addition, we introduce momentum encoders for vision and language modalities respectively (Parameters are )and (Parameters are ), whose parameters are updated based on momentum:

[0177]

[0178] Represents the momentum coefficient, which is used to maintain the consistency of representation in the dynamic repository so that these features are updated smoothly. It is generally taken as a relatively large value, and is set to

[0179] Further, we create two dynamic repositories and Used to store features extracted by the visual momentum encoder And the characteristics of the text momentum encoder N v and N t Represent the size of the visual and text dynamic repositories respectively. In each round of training, the data of the current batch is input into the momentum encoder to obtain the momentum feature representation, and then input into the dynamic repository to update the repository, and the oldest batch of data is removed from the repository. This operation is performed after calculating the loss.

[0180] In order to make full use of the large number of negative samples we mined, we adopt InfoNCE loss as the objective function. Given each image in the data batch, its paired text is regarded as a positive sample and B t All stored samples are regarded as negative samples, and we define the momentum contrast loss (MCL) for image to text as:

[0181]

[0182] Among them, v i Represents the feature vector of the current batch of images, Represents the momentum feature vector of its paired text. The similarity function sim(·,·) uses the dot product of two vectors to calculate their similarity score.

[0183] Similarly, given each text in a batch of data, we define the momentum contrastive loss for text to image as:

[0184]

[0185] Among them, t i Represents the feature vector of the current batch of text, Represents the momentum feature vector of its paired image.

[0186] The final momentum contrast loss is:

[0187]

[0188] 2.4 Instance Neighborhood Semantic Association Modeling

[0189] Samples in the same batch usually share rich semantic information, and effectively utilizing this information is crucial to obtaining discriminative visual-language embeddings. In order to accurately capture the neighborhood semantic associations between instances, we incorporate the semantic knowledge shared by samples into cross-modal feature interactions. Specifically, we construct a multimodal instance neighborhood semantic association graph and apply graph neural networks on this graph for intra-modal and inter-modal semantic interactions.

[0190] 2.4.1 Construction of multimodal instance neighborhood semantic association graph

[0191] First, we construct instance neighborhood semantic association graphs for intra-modality and inter-modality, respectively. For the intra-modality association graph, we use the Euclidean distance of the global visual / textual embedding to quantify the semantic similarity between instances:

[0192]

[0193] in denote the global feature vectors of the i-th and j-th images respectively, They represent the global feature vectors of the i-th and j-th texts respectively, |·|2 represents the l2 norm, and ε>0 is the temperature parameter.

[0194] However, due to the semantic gap between images and texts, directly using global embeddings to construct cross-modal instance neighborhood semantic association graphs may result in loss of semantic information. Inspired by the HREM work [9], we adopt fine matching of local features to measure cross-modal instance neighborhood semantic association. Specifically, given a pair of image-text and their region-word similarity matrices, we find the text word (or visual region) with the highest matching degree for each region (or each word), and then average these maximum matching scores as a measure of cross-modal instance neighborhood semantic association:

[0195]

[0196] Where n r ,n t Represent the number of image regions and the number of text words respectively, They are the τth region features of the ith image and the ηth word features of the jth text, respectively.

[0197] Finally, the multimodal instance neighborhood semantic association graph we obtained can be expressed as:

[0198]

[0199] Where m is the batch size.

[0200] 2.4.2 Instance Neighborhood Semantic Interaction

[0201] After constructing the multimodal instance neighborhood semantic association graph, we apply a multimodal graph neural network to perform instance neighborhood semantic interactions on the graph to capture high-order semantic relationships between instances.

[0202] Specifically, we first introduce the graph convolutional network (GCN) to realize the intra-modal neighborhood semantic interaction of image modality and text modality. GCN can update the embedding of each node by aggregating neighboring nodes. For the image modality, the l-th layer GCN can be expressed as:

[0203]

[0204] in represents the image node feature matrix of the lth layer, Represents the neighborhood semantic association graph of image modality instances, is the learnable weight matrix of layer l, σ(·) is the nonlinear activation function (we use LeakyReLU), are the diagonal elements of the degree matrix.

[0205] Similarly, for the text modality, the l-th layer GCN can be expressed as:

[0206]

[0207] To further capture high-order neighborhood semantic associations between modalities, we use the joint modality GCN to aggregate neighborhood structure knowledge. Specifically, we concatenate the outputs of the image modality GCN and the text modality GCN in the batch dimension, and then input the concatenated features into the joint modality GCN for cross-modal interaction, embedding instance neighborhood correlation into the feature learning process:

[0208]

[0209] in is the joint modality node feature matrix of the lth layer, is a multimodal instance neighborhood semantic association graph, is a learnable weight matrix.

[0210] In order to avoid over-smoothing and introduce noisy semantic associations, we further apply the joint modality graph attention network (GAT) to adaptively and finely capture neighborhood semantic associations. Specifically, we use the output of the joint modality GCN as the input of the joint modality GAT, which can assign different attention weights to different neighbors and mine neighborhood semantic associations more finely.

[0211] In order to further improve the quality of instance neighborhood semantic association, we optimize the original multimodal instance neighborhood semantic association graph S to obtain the optimized adjacency matrix Specifically, we sort the element values ​​in S, and for each submatrix S I→I , S I→T , S T→I and S T→T , we set the elements with values ​​in the last 50% to -∞, and multiply the elements with values ​​in the first 50% by a magnification factor α>1, and get the optimized adjacency matrix S:

[0212]

[0213] Here, Λ(·) represents the median operation. In this way, we retain strong semantic associations while filtering out noisy semantic associations, thereby improving the quality of instance neighborhood semantic associations.

[0214] After obtaining the optimized adjacency matrix After that, we replace the original S with it for instance neighborhood semantic interaction. Specifically, we Incorporated into GAT's self-attention score calculation:

[0215]

[0216] in is the query, key, and value matrix obtained by linearly transforming the output of the joint modality GCN, d k is the dimension of the key vector.

[0217] This method effectively incorporates instance neighborhood semantic association information into the feature learning process, enhancing the model's ability to capture complex semantic relationships. Our method not only considers the semantic association within the modality, but also explores the semantic interaction between modalities, thereby generating more discriminative enhanced visual-language embeddings.

[0218] After instance neighborhood semantic interaction, we obtain the final enhanced visual-language embedding, denoted as and Where m is the batch size.

[0219] To accelerate convergence and further improve the quality of learned embeddings, we adopt the widely used hard negative triplet loss L tri . This ensures that the matching image-text pairs are closer than the hardest matching image-text pair:

[0220] L tri (v,t)=[γ-s(v,t)+s(v,t - )] + +[γ-s(v,t)+s(v - ,t)] + , (26)

[0221] where γ represents the marginal hyperparameter, s(·,·) is used to calculate the cosine similarity of visual-language embeddings, and [·] + ≡max(·,0). (v,t) is a positive image-text pair, v - = argmax x≠v s(x,t) and t - = argmax y≠t s(v,y) are the hardest negative image and negative text in the batch, respectively, used to construct the negative image-text pair (v - ,t) and (v,t - ).

[0222] In order to maintain the speed advantage of the two-stream encoder at the inference stage, we not only calculate the triplet loss on the original visual-language embeddings, but also on the enhanced embeddings and perform cross-alignment. Therefore, the total instance neighborhood semantic interaction loss is defined as follows:

[0223]

[0224] in, Represents the prototype guidance cross-modal alignment loss for the enhanced embedding calculation, which helps to alleviate the problems that may be caused by "soft positive samples" and further improve the performance of the model.

[0225] 3 Objective Function

[0226] The final objective function is:

[0227] L=L PGA +L MCL +L NSI , (28)

[0228] 4 Experimental comparison

[0229] 4.1 Dataset

[0230] This study uses two widely used benchmark datasets for experimental evaluation: Flickr30K

[36] and MS-COCO

[22] . These datasets are widely used in cross-modal retrieval tasks due to their scale and diversity.

[0231] The Flickr30K dataset contains 31,783 images, each with 5 manually annotated description sentences. Following the standard partitioning protocol

[14] , we use the following split: 29,783 images for training, 1,000 for validation, and 1,000 for testing.

[0232] MS-COCO is a larger dataset with 123,287 images, each with 5 descriptive captions. Following previous studies [4]

[27] , we split the dataset into 113,287 images for training, 5,000 for validation, and 5,000 for testing. Following previous practice, we report results on two test settings: (1) 5K test set: directly evaluating on all 5,000 test images; (2) 1K test set: testing on 5 subsets of 1,000 images and taking the average.

[0233] 4.2 Evaluation Metrics

[0234] To comprehensively evaluate the model performance, we use widely used evaluation metrics in image-text retrieval tasks, including Recall@K (R@K) and rSum. R@K calculates the proportion of correct matches in the first K candidates of the retrieval result. We report the results when K = 1, 5, and 10. rSum is the sum of all R@K metrics, calculated as follows:

[0235]

[0236] This comprehensive indicator reflects the overall matching performance of the model in both image-to-text and text-to-image directions. Through these evaluation indicators, we can fully understand the performance of the model in cross-modal image-text matching tasks, including its accuracy at different retrieval depths and the overall effect of bidirectional retrieval.

[0237] 4.3 Implementation Details

[0238] We use the Faster R-CNN model (based on the ResNet-101 backbone network) pre-trained on the Visual Genome dataset to extract image region features. Specifically, we select K=36 region proposals with the highest confidence, and each region is represented by a 2048-dimensional feature vector. At the same time, we use the pre-trained BERT-base model to obtain 768-dimensional word-level features. In addition to local features, we also introduce the CLIP (ViT / B-32) model to extract global representations of images and text. The feature dimension generated by CLIP is 512, which is used to enhance the global semantic understanding of cross-modal matching. In order to fully verify the effectiveness of our method while reducing computing resources, the CLIP model was not fine-tuned in our experiments.

[0239] During training, we use the AdamW optimizer. The initial learning rate is set to 5e-4. The batch sizes for the Flickr30K and MS-COCO datasets are 128 and 256, respectively. The dimension of the joint embedding space is set to 1024, and the momentum coefficient is is set to 0.999, the magnification factor α is set to 1.5, and the margin hyperparameter γ is set to 0.2. The dynamic queue sizes of Flickr30K and MS-COCO are 2048 and 4096, respectively, and the number of prototypes k is 384 and 512, respectively. Our experiments are implemented in the PyTorch framework and trained using a single NVIDIA RTX A6000 GPU.

[0240] Table 1 Comparison with state-of-the-art methods on the Flickr30K and MSCOCO 1k test sets. * indicates the combined results of two models, and the best results are shown in bold.

[0241]

[0242]

[0243] Table 2 Comparison with state-of-the-art methods on the MSCOCO 5k test set. * indicates the ensemble results of two models, and the best result is shown in bold.

[0244]

[0245]

[0246] Table 4 Performance comparison of different dynamic repository sizes on the MSCOCO 1K and MSCOCO 5K test sets

[0247]

[0248] 4.4 Comparison with State-of-the-Art Methods

[0249] To verify the effectiveness of the proposed method, we comprehensively compare our results with the state-of-the-art methods on two benchmark datasets. These methods can be roughly divided into two categories: interactive embedding methods and independent embedding methods.

[0250] Among them, the interactive embedding methods include SCAN, IMRAM, GSMN, SGRAF, UARDA, DREN, NAAF, RCAR, CHAN and BOOM; the independent embedding methods include VSE++, DSRAN, CAMERA, CLIP (ViT / B-32), MEMBER, GPO, DIME, VSRN++, AME, TGDT, HREM, ESA, IMEB and USER. These methods cover the mainstream technologies in the current image-text matching field.

[0251] The method we proposed belongs to the independent embedding class, has good scalability and fast reasoning ability, and integrates the fine-grained interaction module commonly used in interactive embedding, thereby effectively improving the fine-grained matching ability of images and texts. Tables 1 and 2 show the quantitative results of our method on two benchmark datasets in detail. Among them, some integrated methods (marked with ★) improve performance by averaging the similarities of different models.

[0252] It is worth emphasizing that even with only a single model, our method comprehensively outperforms all existing state-of-the-art methods in two key metrics, R@K and rSum, with the only exception being the R@1 metric for text retrieval from image in Table 2. This result fully demonstrates the superior performance and wide applicability of our method.

[0253] We observed impressive results on the Flickr30K dataset. Compared with the state-of-the-art standalone embedding method BOOM and the interactive embedding method USER, our method achieved 21.9% and 10% relative improvements on the rSum metric, respectively. In particular, compared with the strongest competitor USER, our method achieved a comprehensive lead in the R@k (k = 1, 5, 10) metric in both text retrieval and image retrieval tasks. In text retrieval, the improvements were 2.7%, 1.1%, and 0.05%, respectively; in image retrieval, the improvements were 3.4%, 1.4%, and 1.3%, respectively. These results strongly demonstrate the effectiveness and superiority of our method.

[0254] Our method also shows significant performance improvements on the MSCOCO dataset. On the MSCOCO 1K test set, our single model outperforms other single methods and ensemble methods, which is similar to the experimental results of Flickr30K. More notably, our method shows even more significant advantages on the larger and more difficult MSCOCO 5K dataset. Our method significantly outperforms all competing methods on almost all evaluation metrics. This result not only further verifies the effectiveness of our method, but also highlights its robustness and generalization ability when dealing with large-scale datasets. These comprehensive results strongly support the excellent performance of our method in image-text matching tasks, while also highlighting its consistent advantages on datasets of different scales and complexities.

[0255] Our approach achieves significant performance gains on two public image-text matching benchmarks. However, this improvement is not entirely due to the transfer of prior knowledge from the CLIP model. We use a frozen CLIP (ViT / B-32) model without any fine-tuning. Tables 1 and 2 clearly show that this basic CLIP model does not perform well on both benchmarks and does not actually constitute a strong baseline.

[0256] It is worth noting that although our main competitor USER also uses the CLIP model to improve retrieval performance, our method still leads in all evaluation indicators. The significant improvement of our method is mainly attributed to the synergy between various modules. This largely verifies the effectiveness of our proposed method.

[0257] In this way, we not only demonstrate the superior performance of our method, but also emphasize that its effectiveness does not simply rely on the pre-trained model, but stems from our unique module design and overall architecture. This further highlights the innovation and potential value of our method.

[0258] 4.5 Ablation Experiment

[0259] In order to study the proposed method in more depth and verify the effectiveness of each component we proposed, we conducted extensive ablation experiments. The "√" in the table indicates that the component is retained, otherwise it is deleted. In Table 3, "Local" and "Global" refer to the feature extraction method. "GGLP" refers to the global semantics-guided adaptive local feature pooling method module, and "MDM" refers to the momentum dynamic learning repository. "PGA" and "NSI" represent the clustering prototype-guided cross-modal alignment module and instance neighborhood semantic interaction.

[0260] Configuration of our method: We first deeply investigated the impact of feature extraction strategies on model performance, focusing on the independent performance of local features and global features and their complementarity. As shown in Table 3, the model performance is relatively weak when only local features are used. In contrast, the performance is significantly improved when global features are used alone (methods 5-8). It is worth noting that the R@1 index of method 5 in the I->T and T->I directions reaches 84.3% and 69.7%, respectively, which is 2.7 and 7.4 percentage points higher than that of method 1.

[0261] A key finding of this study is the complementarity of local and global features. By introducing the global-local semantics-guided adaptive local feature pooler (GGLP) module, the model performance has achieved a qualitative leap. The GGLP module deeply integrates and utilizes these two complementary features to give full play to their respective advantages. Specifically, the RSUM index of method 10 is improved by 29.3% and 14% respectively compared with the best method using local features alone (method 4) and the best method using global features alone (method 8), which is a remarkable increase.

[0262] Further analysis shows that method 10 has achieved comprehensive improvements in all indicators compared to method 9, which uses a simple weighted addition of global and local features. This strongly confirms the excellent performance of the GGLP module in feature fusion. Local features are good at capturing fine-grained regional information, while global features have advantages in understanding overall semantics. By intelligently integrating these two features, the GGLP module provides the model with a more comprehensive and powerful representation capability, significantly improving the performance of cross-modal retrieval tasks.

[0263] By comparing Method 1 with Method 2, and Method 5 with Method 6, we found that the use of Momentum Dynamic Memory (MDM) brought consistent performance improvements. As a dynamically learned repository, MDM stores a large number of image and text feature representations, which serve as negative samples for contrastive learning and effectively improve the discriminative ability of features. Unlike static storage structures, MDM can continuously update and adjust the stored information during the training process. In the early stages of training, MDM may store more basic cross-modal correspondences; as training deepens, it gradually captures more complex and detailed semantic mappings, thereby improving retrieval accuracy.

[0264] Comparing Method 2 and Method 3, as well as Method 6 and Method 7, we observe that the introduction of the clustered prototype guided cross-modal alignment module (PGA) brings significant improvements in most indicators. When using local features (Method 2 vs Method 3), R@1 in the I->T direction is improved from 81.9% to 82.2%, and in the T->I direction from 62.5% to 62.8%. When using global features (Method 6 vs Method 7), the improvement is more obvious, with R@1 in the I->T direction increasing from 84.6% to 85.1%, and in the T->I direction from 70.0% to 70.5%.

[0265] These results demonstrate the effectiveness of clustering prototypes in guiding cross-modal alignment. PGA provides a common reference point for features of different modalities by generating representative category prototypes. This helps reduce the semantic differences between modalities and enables the model to learn more consistent and robust cross-modal representations. In particular, when dealing with samples with similar semantics but large differences in visual representation, PGA can provide more stable alignment guidance and improve retrieval accuracy.

[0266] In addition, the effect of PGA on global features is more significant, indicating that clustering prototypes have better synergy with global semantic information. This synergy may be due to the fact that global features can provide more comprehensive semantic context, making the generation and application of clustering prototypes more precise and effective.

[0267] By comparing Method 3 and Method 4, as well as Method 7 and Method 8, we can clearly see the performance improvement brought by the NSI module. The core idea of ​​NSI is to use the semantic relationship between instances to enhance feature representation. It captures more detailed semantic structure by considering the neighborhood information of each instance in the feature space. This method is particularly helpful for processing samples that are similar in global semantics but different in details.

[0268] The significant improvement of NSI in the R@10 indicator highlights its advantage in improving the comprehensiveness of retrieval. NSI can effectively capture instances that are semantically similar but not identical, expanding the scope of retrieval while maintaining high relevance. This feature is particularly important when dealing with complex cross-modal retrieval tasks, helping the model to establish more detailed and coherent mapping relationships in the semantic space.

[0269] It is worth noting the synergy between NSI and other modules such as PGA. PGA provides category-level alignment guidance, while NSI performs refinement at the instance level. This multi-scale semantic interaction strategy enables the model to take into account both macro-level category information and micro-level instance features, thus performing well in various complex retrieval scenarios.

[0270] Size of the Momentum Dynamic Memory (MDM): To further evaluate the impact of the size of the Momentum Dynamic Memory (MDM) on the retrieval performance, we conducted experimental evaluations on the MSCOCO 5K test set and the MSCOCO 1K test set.

[0271] The results in Table 5.4 show that MDM size has a significant impact on model retrieval performance. As the MDM size increases from 256 to 2048, the Rsum indicators of both test sets show a continuous upward trend, with the 1K test set increasing by 1.14% and the 5K test set increasing by 2.73%. This shows that a larger repository can accommodate more diverse negative samples, which helps the model learn more robust feature representations. However, when the MDM size is further increased to 4096, the performance decreases slightly, probably due to the introduction of too much noise. It is worth noting that the change in MDM size has a more significant impact on the 5K test set, which may suggest that larger MDMs have advantages when dealing with larger or more complex datasets.

[0272] Hyperparameters: \figref{fig:Parameter analysis k}, \figref{fig:Parameter analysis alpha} show the impact of two key parameters on retrieval performance, where k is the number of cluster prototypes and α is the semantic magnification factor of the multimodal instance neighborhood semantic association graph. It can be seen that the model performance remains relatively stable under these parameter changes, indicating good hyperparameter robustness.

[0273] Table 5 Performance comparison of models trained on MSCOCO and evaluated on the Flickr30K test set

[0274]

[0275] 4.6 Further Analysis

[0276] Generalization ability: To further verify the generalization ability of the model, we conducted a zero-shot transfer experiment across datasets. Specifically, we tested the performance of the model trained on the MSCOCO dataset on the Flickr30K dataset. As shown in Table 5, our method shows excellent performance in this task, proving that our model can effectively capture common semantic relationships across datasets, highlighting the superiority of our proposed method.

[0277] Retrieval efficiency:

[0278] Figure 4 It is the parameter analysis of Equation 8 on two data sets;

[0279] Figure 5 is the parameter analysis of Eq. 21 on two data sets; Figure 6The performance and computation time comparison of cross-modal retrieval using RTX3090 GPU on the Flickr30K test set is shown. Our method achieves an excellent balance between accuracy and efficiency, significantly outperforming other methods while maintaining a very low average retrieval time. Compared with embedding-based methods (such as IMEB, HREM), we have achieved significant performance improvements; compared with interaction-based methods (such as CHAN, NAAF), the computation time has been greatly reduced. This result highlights the advantages of our method in efficient and accurate retrieval, especially for large-scale dataset scenarios.

[0280] 5 Conclusion

[0281] In this study, we focus on a core issue that is often overlooked in image-text matching: high-order associations and subtle differences between semantically similar instances. Existing methods often mislabel semantically similar instances during dataset annotation, forming so-called "soft positive samples", which introduce additional noise and confusion in model training. In addition, during the inference phase, the model is also prone to mistakenly identify instances with good local segment alignment but global semantic mismatch as positive samples, namely "soft negative samples", which significantly reduces the matching accuracy. At the same time, we also noticed that there are a large number of semantically similar instances in the training batch, which have high-order shared knowledge. To this end, we propose a fully-granular prototype-guided neighborhood embedding framework (FPNE), which effectively captures high-order associations and subtle differences between instances. Extensive experiments demonstrate the significant advantages of our method.

Claims

1. A clustering prototype-guided aggregated neighborhood embedding image-text matching method, characterized in that: The following steps are involved: S1. Construct an aggregated full-granularity semantic representation learning network, including joint extraction of global and local features, adaptive local feature aggregation guided by global semantics, and full-granularity feature gating fusion; S2, using cluster prototype momentum contrast learning network, through online clustering, dynamic learning of unified cluster prototypes in the joint representation space of images and texts; S3, image and text features are projected into the shared prototype space, and the similarity with each prototype is calculated; S4. Use the Sinkhorn algorithm to calculate the soft assignment probability distribution and use the cross entropy loss as the comparative learning target; S5, introduces two momentum encoders to generate a large dynamic repository as additional negative samples; S6. Instance neighborhood relationship modeling uses global and local features to construct intra-modal and inter-modal correlation matrices to characterize the semantic similarity between instances.

2. The clustering prototype-guided aggregated neighborhood embedding image-text matching method according to claim 1, characterized in that: In S1: The specific steps of joint extraction of global and local features are as follows: for the input image I, the bottom-up attention model [1] is used to extract the first K = 36 most significant regional features using the Faster R-CNN network pre-trained on the Visual Genomes dataset. Each regional feature is mapped to a d-dimensional joint semantic space through a fully connected linear projection layer to obtain the image region representation. Where n r is the number of regional features; For the input sentence T, the pre-trained BERT model is used as the text encoder to extract word-level feature representations, and then a fully connected linear projection layer is used to map these word features into the same d-dimensional joint semantic space as the image, resulting in Where n t is the number of word-level features; Fine-grained semantic association feature enhancement is performed for visual regions and text words. Taking the image modality as an example, a semantic relationship graph between visual regions in an image is constructed, and a graph attention network is proposed to learn the contextual semantics of regional features. The nodes of the graph are sets of regional features, and the edges represent the semantic associations between regions. Original image region representation After the graph attention network, we obtain fine-grained semantic association enhanced regional features Similarly, we can obtain fine-grained semantically related enhanced word features The pre-trained CLIP model is introduced to extract global feature representation, and the CLIP (ViT / B-32) model is directly used to extract visual semantic embedding without fine-tuning; through the fully connected linear projection layer, we obtain the global feature representation and v of images and text g ∈R d , t g ∈R d Together with the regional features and word features, they are located in the d-dimensional joint semantic space.

3. The clustering prototype-guided aggregated neighborhood embedding image-text matching method according to claim 2, characterized in that: In S1: The adaptive local feature aggregation guided by global semantics is as follows: The input image is passed through the feature extraction network to obtain the local feature vector as the query, and the image is divided into several salient regions. The feature vector of each region is regarded as a codeword to form the codebook of the image. Perform multiple linear transformations on the query and codebook, and calculate the similarity between the transformed query and codeword to obtain multiple sets of soft-assigned coding coefficient vectors; The vector is then mean-pooled to obtain the final coding coefficient vector; Finally, through softmax normalization, this vector is converted into a set of fine-grained adaptive weights, which are used to perform weighted pooling on the features of the salient regions to obtain the encoded representation of the image.

4. The clustering prototype-guided aggregated neighborhood embedding image-text matching method according to claim 3, characterized in that: In S1: Capture subtle semantic differences between samples through cluster mapping, and then compare them in the cluster semantic space; Define a set of trainable prototype vectors P=p1,p2,…,p in the joint representation space of image and text k , where k is the number of prototypes; these prototype vectors can be regarded as low-dimensional projections of the entire data set, storing frequent, common, and transferable features in the data set; Given a pair of image embeddings V = v1,…,v m and text embedding T = t1,…,t m , project it into the unified prototype space, and get the similarity scores of the image and text with each prototype: in Each row represents the similarity score between a sample and k prototypes; Use the Sinkhorn algorithm to get the soft assignment matrix of image and text from the similarity scores Each sample is softly assigned to multiple prototypes, and the sum of the assignment probabilities is 1; the Sinkhorn algorithm assigns u v and u t Convert to soft allocation matrix; Compute the prototype-guided cross-modal alignment loss (PGA) of image perspective and text perspective; For each image view, the soft assignment matrix D of the text embedding t As the "positive sample" label, the similarity score u between the image and the prototype v As the predicted value, negative sampling comparison is performed; The goal is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs; minimize the negative cross entropy loss between the two, that is, maximize the similarity of positive sample pairs, and naturally minimize the similarity of negative sample pairs. The contrast loss of image perspective can be calculated by the following formula: Where m is the batch size, k is the number of prototypes, τ is the temperature coefficient, represents the probability that the i-th text embedding is assigned to the j-th prototype, represents the similarity score between the i-th image embedding and the j-th prototype; Similarly, for the text perspective, the soft assignment matrix D of the image embedding is v As the "positive sample" label, the similarity score u between the text and the prototype t As the predicted value, negative sampling comparison is performed, and the contrast loss of the text perspective can be calculated by the following formula: The final prototype-guided cross-modal alignment loss is the sum of the image perspective and text perspective losses:

5. The clustering prototype-guided aggregated neighborhood embedding image-text matching method according to claim 4, characterized in that: It also includes building two dynamic repositories to decouple the repository size from the mini-batch size, so the size of negative samples can be much larger than the mini-batch size; Let θ v With θ t They are the visual encoder f v and the text encoder f t The parameters of the momentum encoder are introduced for the visual and language modalities respectively. (Parameters are )and (Parameters are ), whose parameters are updated based on momentum: represents the momentum coefficient, which takes a relatively large value and is set in this paper as Create two dynamic repositories and Used to store features extracted by the visual momentum encoder j∈[1,N v ] and the characteristics of the text momentum encoder j∈[1,N t ],N v and N t Respectively represent the The size of the dynamic repository of perception and text. In each round of training, the data of the current batch is input into the momentum encoder to obtain the momentum feature representation, and then input into the dynamic repository to update the repository, and the oldest batch of data is removed from the repository. This operation is performed after calculating the loss. Using InfoNCE loss as the objective function, given each image in the data batch, its paired text is regarded as a positive sample, and B t All stored samples are regarded as negative samples, and we define the momentum contrast loss (MCL) for image to text as: Among them, v i Represents the feature vector of the current batch of images, Represents the momentum feature vector of its paired text. The similarity function sim(·,·) uses the dot product of two vectors to calculate their similarity score; Similarly, given each text in a batch of data, we define the momentum contrastive loss for text to image as: Among them, t i Represents the feature vector of the current batch of text, represents the momentum feature vector of its paired image; The final momentum contrast loss is:

6. The clustering prototype-guided aggregated neighborhood embedding image-text matching method according to claim 1, characterized in that: In S6, each instance and its related neighbors are constructed as a neighborhood subgraph, and a graph neural network is introduced to perform modal interactive learning to fully explore the neighborhood relationship of the sample instance; Incorporate the semantic knowledge shared by samples into cross-modal feature interactions, construct a multimodal instance neighborhood semantic association graph, and apply graph neural networks on the graph to perform semantic interactions within and between modalities. The instance neighborhood semantic association graphs are constructed within and between modalities respectively. The intra-modal association graph uses the Euclidean distance of the global visual / text embedding to quantify the semantic similarity between instances: in denote the global feature vectors of the i-th and j-th images respectively, Respectively represent the global feature vectors of the i-th and j-th texts, and |·|2 represents norm, ε>0 is the temperature parameter; The fine matching of local features is used to measure the semantic association of cross-modal instance neighborhoods. Specifically, given a pair of image-text and its region-word similarity matrix, the text word (or visual region) with the highest matching degree is found for each region (or each word), and then these maximum matching scores are averaged as the measure of the semantic association of cross-modal instance neighborhoods: Where n r ,n t Represent the number of image regions and the number of text words respectively, are the τth region feature of the ith image and the ηth word feature of the jth text respectively; The final multimodal instance neighborhood semantic association graph can be expressed as: Where m is the batch size; After constructing the multimodal instance neighborhood semantic association graph, a multimodal graph neural network is applied to perform instance neighborhood semantic interaction on the graph. The graph convolutional network (GCN) is introduced to realize the intra-modal neighborhood semantic interaction of image modality and text modality. For the image modality, the l-th layer of GCN can be expressed as: in represents the image node feature matrix of the lth layer, Represents the neighborhood semantic association graph of image modality instances, is the learnable weight matrix of layer l, σ(·) is the nonlinear activation function, are the diagonal elements of the degree matrix; Similarly, for the text modality, the l-th layer GCN can be expressed as: Use the joint modality GCN to aggregate neighborhood structure knowledge, concatenate the outputs of the image modality GCN and the text modality GCN in the batch dimension, and then input the concatenated features into the joint modality GCN for cross-modal interaction, embedding instance neighborhood correlation into the feature learning process: in is the joint modality node feature matrix of the lth layer, is a multimodal instance neighborhood semantic association graph, is a learnable weight matrix; A joint-modality graph attention network (GAT) is applied to adaptively and finely capture neighborhood semantic associations, and the output of the joint-modality GCN is used as the input of the joint-modality GAT. Optimize the original multimodal instance neighborhood semantic association graph S to obtain the optimized adjacency matrix Sort the element values ​​in S, for each submatrix S I→I , S I→T , S T→I and S T→T , set the elements with values ​​in the last 50% to -∞, and multiply the elements with values ​​in the first 50% by a magnification factor α>1, and get the optimized adjacency matrix S: Among them, Λ(·) represents the operation of taking the median; After obtaining the optimized adjacency matrix After that, it replaces the original S for instance neighborhood semantic interaction. Incorporated into GAT's self-attention score calculation: in is the query, key, and value matrix obtained by linearly transforming the output of the joint modality GCN, d k is the dimension of the key vector; After instance neighborhood semantic interaction, the final enhanced visual-language embedding is obtained, denoted as and Where m is the batch size; Adopt hard negative triplet loss L tri , ensuring that the distance between matching image-text pairs is closer than the distance between the most difficult image-text pair to match: L tri (v,t)=[γ-s(v,t)+s(v,t - )] + +[γ-s(v,t)+s(v - ,t)] + , where γ represents the marginal hyperparameter, s(·,·) is used to calculate the cosine similarity of visual-language embeddings, and [·] + ≡max(·,0), (v,t) is a positive image-text pair, v - = argmax x≠v s(x,t) and t - = argmax y≠t s(v,y) are the hardest negative image and negative text in the batch, respectively, used to construct the negative image-text pair (v - ,t) and (v,t - ); Not only the triplet loss is calculated for the original visual-language embedding, but also the loss is calculated for the enhanced embedding and cross-aligned. The total instance neighborhood semantic interaction loss is defined as follows: in, Represents the prototype-guided cross-modal alignment loss for the enhanced embedding calculation, and the final objective function is: L=L PGA +L MCL +L NSI ,.

Images