A semantically anchored guided approximate contrastive learning method and system

Abstract

This invention discloses a semantically anchored, approximate contrastive learning method and system, relating to the fields of vision and language technology. It involves acquiring image and text data; extracting image embedding vectors and text embedding vectors using an image encoder and a text encoder, respectively; updating dynamic image and text queues through cross-modal momentum contrast and constructing a fused multimodal representation, which serves as a shared semantic anchor; progressively calibrating the image and text embedding vectors towards the shared semantic anchor within the contrastive learning framework to generate calibrated image and text embedding vectors; and performing image-text retrieval based on the calibrated image and text embedding vectors. This invention relaxes the strict requirement for complete semantic alignment of positive sample pairs, effectively mitigating cross-modal semantic bias.

Description

Technical Field

[0001] This invention relates to the fields of vision and language technology, and more specifically to a semantically anchored guided approximate contrastive learning method and system. Background Technology

[0002] With the explosive growth of multimodal data, image-text retrieval has become a fundamental task connecting the fields of vision and language. Its core objective is to learn a shared embedding space to map the embedding vectors of positive sample image-text pairs to each other, while repelling the embedding vectors of negative sample image-text pairs. This enables the retrieval of relevant images for a given text query or vice versa, which can be viewed as a contrastive learning problem.

[0003] However, existing contrastive learning methods are generally based on a strong assumption: that every image-text pair in the training data is semantically perfectly aligned. This assumption is often difficult to apply in practice, mainly because it leads to a rigid alignment target, treating all positive sample pairs as equally reliable while ignoring the inherent uncertainties in real-world annotations. In fact, even high-quality benchmark datasets like Flickr30K and MS-COCO contain significant semantic biases. For example, text may omit parts of visual objects in an image (such as "black bag" or "tree"), or contain descriptions that are not entirely consistent with the image content. This bias means that "positive sample pairs" are not semantically perfectly aligned, and traditional contrastive learning methods, by forcing alignment, introduce incorrect supervisory signals.

[0004] Furthermore, traditional contrastive learning methods require direct alignment of image and text embedding vectors. Direct alignment of image and text embeddings is often affected by modal gaps and sensitivity to title noise—especially when the paired data contains vague or inaccurate descriptions. In summary, there is an urgent need for a more robust and semantically consistent cross-modal alignment mechanism to improve retrieval performance and enhance the model's adaptability to imperfect data in real-world scenarios. Summary of the Invention

[0005] In view of this, the present invention provides a semantically anchored guided approximate contrastive learning method and system to solve the problems in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: On one hand, this invention discloses an approximate contrastive learning method guided by semantic anchoring, comprising: Acquire image and text data; Image embedding vectors and text embedding vectors are extracted using an image encoder and a text encoder, respectively. By cross-modal momentum comparison, dynamic image queues and dynamic text queues are updated, and fused multimodal representations are constructed as shared semantic anchors; In the contrastive learning framework, the image embedding vector and the text embedding vector are progressively calibrated toward the shared semantic anchor to generate calibrated image embedding vectors and calibrated text embedding vectors. Image-text retrieval based on calibrated image embedding vectors and text embedding vectors.

[0007] Through the above technical solution, the technical effect of this invention is as follows: by constructing a fused multimodal representation as a shared semantic anchor, the decoupling alignment target of image embedding vectors and text embedding vectors is changed from the traditional direct pairwise alignment to convergence towards the shared anchor. Simultaneously, a cross-modal momentum contrast mechanism is introduced to dynamically maintain the negative sample queue, achieving progressive calibration within the contrastive learning framework.

[0008] Preferably, in the semantically anchored guided approximate contrastive learning method described above, the specific steps of cross-modal momentum contrast are as follows: A cross-modal momentum comparison mechanism is adopted to maintain a dynamic image queue and a dynamic text queue. The dynamic image queue stores historical image embedding vectors generated by the key image encoder, and the dynamic text queue stores historical text embedding vectors generated by the key text encoder. The dynamic image queue and dynamic text queue are used as negative sample sets for comparative learning. The parameters of the key image encoder are updated based on the exponential moving average of the query image encoder parameters, and the parameters of the key text encoder are updated based on the exponential moving average of the query text encoder parameters.

[0009] The technical effects of the present invention through the above technical solution are as follows: A momentum-updated key encoder is used to generate historical embeddings, and dynamic image and text queues are maintained. Query encoder parameters are optimized through backpropagation, and key encoder parameters are updated based on the exponential moving average of query encoder parameters, ensuring the diversity and stability of negative samples in the queues. This significantly expands the negative sample size without increasing computational cost, enhancing the discriminative ability of contrastive learning and avoiding model degradation caused by insufficient negative samples in traditional methods.

[0010] Preferably, in the above-mentioned semantically anchored approximate contrastive learning method, the specific steps for constructing a fused multimodal representation are as follows: Image region features and text lexical features are projected into a shared joint embedding space, respectively. The projected image features are concatenated with the text features to form a joint feature sequence; The joint feature sequence is input into the Transformer encoder to model the semantic interactions within and between modes; The generalized pooling operator aggregates the interacting features into a fused multimodal representation.

[0011] The technical effect of this invention through the above technical solution is as follows: Image region features and text lexical features are projected onto a shared joint embedding space and then concatenated to form a joint feature sequence. A Transformer encoder models the semantic interactions within and between modalities, and finally, a generalized pooling operator aggregates them into a fixed-dimensional fused multimodal representation. This fused multimodal representation simultaneously contains complementary information from both images and text, and as a semantic anchor, it possesses stronger semantic expressive power, providing a high-quality target reference for subsequent single-modal embedding calibration.

[0012] Preferably, in the above-mentioned semantic anchoring-guided approximate contrastive learning method, the specific steps for gradually calibrating the image embedding vector and text embedding vector toward the shared semantic anchor point are as follows: using an iterative optimization method, the image embedding vector and text embedding vector are gradually approximated toward the fused multimodal representation; In each iteration, the image embedding vector and the text embedding vector are updated by a calibration function so that the image embedding vector and the text embedding vector are semantically close to the shared semantic anchor point.

[0013] The technical advantages of this invention through the above-described technical solution are as follows: By employing an iterative optimization approach, the image embedding and text embedding are updated separately in each iteration using a calibration function, gradually approaching a fused multimodal representation. This process allows each modality to progressively remove modality-specific features while retaining semantic information consistent with multimodal consensus, achieving a gradual and stable convergence process. It avoids the problem of residual modality-specific information that may result from single-step calibration, thus improving the semantic consistency of the calibrated embeddings.

[0014] Preferably, in the above-mentioned semantically anchored guided approximate contrastive learning method, the training objectives within the contrastive learning framework include: The first loss function is the hub-aware loss function, which is used to align the query image embedding and the query text embedding in the joint embedding space within a single batch, thereby suppressing the hub problem in the high-dimensional embedding space. The second loss function is used to align the query image embedding with the key text embedding in the dynamic text queue and the query text embedding with the key image embedding in the dynamic image queue in cross-modal momentum comparison. A third loss function is used to align the query image embedding with the shared semantic anchor and to align the query text embedding with the shared semantic anchor. The overall loss function is a weighted sum of the first loss function, the second loss function, and the third loss function, and the weights of the first loss function and the third loss function are controlled by hyperparameters.

[0015] Through the above technical solution, the technical effects of this invention are as follows: A triple loss function is constructed: the first loss employs hub-aware loss to suppress hub problems in high-dimensional space; the second loss aligns query embeddings with key embeddings in the dynamic queue through cross-modal momentum comparison; and the third loss aligns query embeddings with shared semantic anchors. The three losses are weighted and summed to form the overall loss. Multi-objective collaborative optimization, acting on different semantic levels, forms a complementary and enhanced optimization mechanism, comprehensively improving the quality of cross-modal representations and retrieval performance.

[0016] Preferably, in the semantically anchored guided approximate contrastive learning method described above, the image encoder includes: The region-level feature extraction module is used to extract multiple region-level visual features of an image through a bottom-up attention mechanism; The global feature extraction module is used to extract global visual features of an image using a pre-trained visual model. The feature enhancement module is used to project the global visual features onto the same feature space as the regional visual features, calculate the attention weights between the global features and each regional feature, and modulate and enhance the regional visual features based on the attention weights to generate enhanced regional features. The pooling module is used to aggregate the enhanced regional features into an image embedding using a generalized pooling operator.

[0017] Through the above technical solution, the technical effects of this invention are as follows: the regional feature extraction module extracts local regional features through Faster R-CNN; the global feature extraction module extracts global semantic features through CLIP; the feature enhancement module calculates the attention weights of global features and regional features, and modulates and enhances the regional features; the pooling module aggregates into image embeddings through generalized pooling operators; and the introduction of globally semantically guided regional feature enhancement enables the image encoder to better capture visual regions related to text descriptions, thereby improving the semantic expressive power of image embeddings.

[0018] Preferably, in the above-described semantically anchored guided approximate contrastive learning method, the text encoder includes: The lexical-level feature extraction module is used to extract multiple lexical-level features from the text using a pre-trained language model; The projection module is used to project the word-level features into the joint embedding space through a fully connected layer; The pooling module is used to aggregate projected word-level features into text embeddings using a generalized pooling operator.

[0019] Through the above technical solution, the technical effect of the present invention is as follows: the word-level feature extraction module extracts word-level features of the text through BERT-base; the projection module maps word-level features to the joint embedding space; the pooling module aggregates into text embeddings through generalized pooling operators, the text encoder can fully capture the semantic information of the text, and achieve cross-modal semantic alignment coordination by sharing the joint embedding space with the image encoder.

[0020] On the other hand, this invention discloses a semantically anchored guided approximate contrastive learning system, which applies the above method and includes: An image encoder is used to extract image embedding vectors from image data; A text encoder is used to extract text embedding vectors from text data; The cross-modal interaction module is used to construct a fusion of multimodal representations as shared semantic anchors; The calibration module is used to progressively calibrate the image embedding vector and text embedding vector towards the shared semantic anchor in the contrastive learning framework, generating calibrated image embedding vector and calibrated text embedding vector. The retrieval module is used for image-text retrieval based on calibrated image embedding vectors and text embedding vectors.

[0021] Through the above technical solution, the technical effect of the present invention is that it provides a complete image-text retrieval system that can effectively handle semantic deviation problems in real-world scenarios and achieves retrieval performance superior to the most advanced existing methods in multiple benchmark tests.

[0022] Preferably, in the above-described semantically anchored approximate contrastive learning system, the cross-modal interaction module includes: Transformer encoders are used to model intramodal and intermodal interactions; Generalized pooling operators are used to aggregate multimodal features into a unified fused multimodal representation. It also includes a cross-modal momentum comparison module for maintaining a dynamic queue to store negative sample embedding vectors and updating key encoder parameters.

[0023] Through the above technical solution, the technical effects of this invention are as follows: The cross-modal interaction module uses a Transformer encoder to model semantic interactions within and between modalities, and aggregates them into a fused multimodal representation through a generalized pooling operator. The cross-modal momentum comparison module maintains a dynamic queue to store negative samples and updates the key encoder parameters. The interaction module and the cross-modal momentum comparison module work together to ensure the semantic richness of the fusion anchor points and enhance the diversity of negative samples in the contrastive learning, forming a complete technical closed loop.

[0024] Preferably, in the above-mentioned semantically anchored approximate contrastive learning system, the cross-modal momentum contrastive module includes: The dynamic queue module is used to maintain dynamic image queues and dynamic text queues, and to store historical embedding vectors generated by the key encoder; The momentum update module is used to update the parameters of the key encoder based on the parameters of the query encoder.

[0025] Through the above technical solution, the technical effects of this invention are as follows: the dynamic queue module maintains a dynamic image queue and a dynamic text queue, and stores the historical embedding vectors generated by the key encoder; the momentum update module updates the parameters of the key encoder according to the parameters of the query encoder, and adopts an exponential moving average strategy; it realizes the smooth update of the key encoder, ensures the stability and diversity of negative samples in the queue, provides a high-quality negative sample set for contrastive learning, and improves the convergence speed and final performance of model training.

[0026] As can be seen from the above technical solution, compared with the prior art, this invention discloses a semantically anchored guided approximate contrastive learning method and system. It replaces the traditional direct alignment of images and text in contrastive learning by constructing fused multimodal representations as shared semantic anchors. Specifically, this method uses a Transformer encoder and a generalized pooling operator to fuse image and text features into a joint semantic anchor. A progressive calibration mechanism is employed to gradually bring the image and text embedding vectors closer to this anchor during iterative optimization. Simultaneously, cross-modal momentum contrast is combined to enhance the diversity of negative samples. Finally, a fusion-aware loss function is used to achieve alignment between single-modal embeddings and multimodal semantic anchors, thereby relaxing the strict requirements for complete semantic alignment of positive samples and effectively mitigating cross-modal semantic bias. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0028] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This invention demonstrates the retrieval performance under different loss weight parameters. Figure 3 This is a visualization of the image-to-text retrieval results of the present invention. Figure 4 This is a visualization of the text-to-image retrieval results of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] This embodiment provides an approximate contrastive learning method guided by semantic anchoring, such as Figure 1 As shown, the specific steps include: Step 1: Obtain image data and text data.

[0031] In this embodiment, the image data is a natural image, and the text data is a natural language description corresponding to the image. The image data can be acquired through an image acquisition device or read from a public dataset, and the text data can be manually annotated or obtained from a dataset.

[0032] Step 2: Extract the image embedding vector and text embedding vector respectively using the image encoder and text encoder.

[0033] The image encoder employs a bottom-up attention mechanism based on Faster R-CNN to extract image region-level features, while simultaneously using CLIP (ViT-B / 32) to extract global visual features. A semantic enhancement module then fuses and modulates the global and region features to generate an enhanced region feature sequence. Subsequently, the generalized pooling operator (GPO) aggregates these features into a fixed-dimensional image embedding vector.

[0034] Specifically, to ensure the consistency of feature representation and to achieve momentum-based parameter updates, the query image encoder and the key image encoder adopt the same architecture; Given an input image, extract using a bottom-up attention mechanism. Each region-level visual feature is implemented using Faster R-CNN: ; in This represents the dimension of each region-level visual feature.

[0035] In addition, global visual features, which include location encoding, are extracted using CLIP (ViTB / 32). ; in, This indicates the dimension of the global image representation.

[0036] Semantic enhancement, achieved through the global representation (Segr) module, improves expressiveness by adjusting the combination of regional features and global semantic information.

[0037] First, global features Through alternating embedding of fully connected (FC) layers, GELU activation functions, and batch normalization layers, the features are projected into the same feature space as the region features. ; in and It is a learnable weight matrix. and It is a learnable bias term. and This represents the batch normalization function. This represents the GELU activation function.

[0038] Calculate the global-to-region attention weights, which are used to measure the relevance of each region's features to the global semantic representation: ; in, Indicates the first Normalized attention weights for each region.

[0039] Subsequently, the resulting attention weights are used to modulate and enhance the features of each region: ; ; Subsequently, the enhanced region features are projected into the joint embedding space through a fully connected layer: ; in, This represents the learnable weight matrix. This represents a learnable bias term.

[0040] Subsequently, the predicted features are aggregated using the generalized pooling operator (GPO). This plug-and-play aggregation module automatically integrates features into a holistic embedding: ; in =1024 indicates the dimension of the shared image-text joint embedding space.

[0041] No. The outputs of the query image encoder and the key image encoder for the image are denoted as image embedding vectors, respectively. and .

[0042] The text encoder uses BERT-base to extract word-level features from the text, projects them into the joint embedding space through a fully connected layer, and finally aggregates them into text embedding vectors through GPO.

[0043] Similar to the image encoder, the query text encoder and the key text encoder use the same architecture; therefore, the query text encoder will be described as a representative example.

[0044] Given a containing The text input consists of 100 words. First, word-level features are extracted using BERT-base. This model internally includes positional encoding. ; in =768 represents the dimension of each word-level feature.

[0045] These features are then projected into the joint embedding space through a fully connected layer:

[0046] in, This represents the learnable weight matrix. This represents a learnable bias term.

[0047] The predicted word-level features are then aggregated using GPO to obtain the final text embedding: ; For the For each text, the query text encoder and the key text encoder generate text embedding vectors respectively. and .

[0048] Step 3: Update the dynamic image queue and dynamic text queue through cross-modal momentum comparison, and construct a fused multimodal representation as a shared semantic anchor.

[0049] In the cross-modal momentum comparison mechanism, two dynamic queues are maintained: a dynamic image queue to store historical image embedding vectors generated by the key image encoder, and a dynamic text queue to store historical text embedding vectors generated by the key text encoder. The parameters of the key encoder are updated by querying the exponential moving average of the encoder parameters. ; ; in, The momentum coefficient; and These represent the parameters of the key image encoder and the key text encoder, respectively. and These represent the parameters for querying the image encoder and querying the text encoder, respectively. The dynamic visual and text queues maintain a fixed-size embedding buffer generated by the key encoder.

[0050] Specifically, the visual queue stores the most recent key image embeddings: ; The text queue stores the latest key text embeddings: ; in K This indicates the capacity of the dynamic queue. New embeddings are added to the queue, while the oldest embeddings are removed, thus maintaining a dynamic queue with a fixed capacity. These queues provide a rich and diverse set of negative samples in mini-batch data, effectively improving the stability and performance of contrastive learning.

[0051] The construction method of fused multimodal representation is as follows: image region features and text lexical features are projected onto a shared joint embedding space and then concatenated to form a joint feature sequence. This sequence is then input into the Transformer encoder to model the semantic interactions within and between modalities. Finally, the sequences are aggregated into a fused multimodal representation using a generalized pooling operator. , as a shared semantic anchor.

[0052] Furthermore, a multimodal information fusion module was designed to unify image and text features into a multimodal representation. Visual features Text features First, it is projected into a shared joint embedding space through a fully connected layer: ; ; in and It is a learnable weight matrix. and It is a learnable bias term; The predicted visual and textual features are then concatenated to form a unified multimodal representation: ; Will The input is fed into the Transformer encoder, which consists of a single-layer structure containing 16 attention heads used to model intra-modal and inter-modal interactions. ; ; ; in , , It is the first Learnable projection matrix for each attention head It is the output projection matrix. Indicates the number of attention heads. This represents the dimension of each attention head.

[0053] The attention output is combined with the input via a residual connection, followed by layer normalization. ; Then a two-layer position feedforward network is applied: ; in and It is a learnable weight matrix. and It is the corresponding bias vector.

[0054] Apply residual connection followed by layer normalization: ; Finally, GPO aggregates the multimodal features into a single fusion vector: ; For clarity, the above describes the fusion process for a single image-text pair. In practical applications, these operations are applied in parallel to a batch of image-text pairs during training. Indicates the first A fused vector of image-text pairs.

[0055] Step 4: In the contrastive learning framework, the image embedding vector and the text embedding vector are progressively calibrated toward the shared semantic anchor to generate calibrated image embedding vectors and calibrated text embedding vectors.

[0056] The goal is to bring matching image-text pairs closer together in the joint embedding space, push away mismatched pairs, and align each pair with its multimodal fusion representation. First, a pivotality-aware loss (HAL) is employed. This method alleviates the pivotality problem in the high-dimensional embedding space by appropriately weighting positive and negative sample pairs, thereby achieving alignment of image-text representations.

[0057] In single-batch training, the loss function is defined as follows:

[0058] in It's a temperature parameter. It is the interval parameter. This refers to the batch size. The cosine similarity is used for the similarity of all loss functions. Perform the calculation.

[0059] In dynamic queue training, the loss function is defined as follows: ; Specifically Encourage each query image embedding vector Close to the positive text embedding vector from the key encoder At the same time, it avoids embedding large amounts of negative example text stored in dynamic queues:

[0060] in K Indicates the size of the dynamic queue.

[0061] Similarly, Embed the query text into a vector Embedded vectors of positive key images Alignment is performed, and the images are compared with all negative image embeddings in the visual queue:

[0062] To ensure that image and text embeddings are consistent with their fused multimodal representations, the pivotal perception loss is extended to a fusion-aware alignment objective. This objective prompts each unimodal embedding to be semantically closer to its corresponding fused multimodal representation, thereby reducing cross-modal bias.

[0063] ; Here, Embed each image vector Its corresponding fusion vector Align and simultaneously separate the fused vectors from other image-text pairs:

[0064] Text fusion loss It is symmetric, embedding each text into a vector. Its corresponding fusion multimodal representation Align it while separating it from other image-text pairs with the fusion vector:

[0065] The overall loss function integrates all objectives: ; in and It is a hyperparameter that controls the relative contribution of each loss term to the overall objective function.

[0066] Step 5: Perform image-text retrieval based on the calibrated image embedding vector and text embedding vector.

[0067] During the inference phase, only the query image encoder and query text encoder are used to extract the embedding vectors of the image and text, and the matching degree between the image and text is calculated by cosine similarity, returning the retrieval result with the highest similarity.

[0068] Based on this, specific applications are as follows: Dataset and implementation details: Experiments were conducted on two widely used benchmark datasets: Flickr30K (31,000 images: 29,000 for training, 1,000 for validation, and 1,000 for testing) and MS-COCO (123,287 images: 113,287 for training, 5,000 for validation, and 5,000 for testing), with each image labeled with five titles. For MS-COCO, results are reported on average performance across the full 5K test set and five 1K test splits, following standard evaluation protocols. Retrieval performance was evaluated using R@K (K=1,5,10), a metric that measures the proportion of correct items in the query terms ranked in the top K search results, following previous research. Furthermore, RSUM, defined as the sum of R@K values ​​for both retrieval directions (image-to-text and text-to-image), was reported, providing an aggregate metric for overall retrieval effectiveness.

[0069] All hyperparameters were determined based on verified performance. Temperature scaling was set for Flickr30K. α =90, boundary β =0.5, loss weight η =20; Set for MS-COCO η =1, Loss Weight θ =0.008, key encoder momentum coefficient m =0.999. The dynamic queue sizes were 2048 for Flickr30K and 4096 for MS-COCO. The AdamW optimizer was used with a weight decay of 10⁻⁴. Flickr30K was trained for 20 epochs with an initial learning rate of 4 × 10⁻⁴, decaying by a factor of 10 in the 10th epoch; MS-COCO was trained for 25 epochs with an initial learning rate of 4 × 10⁻⁴, decaying by a factor of 10 in the 15th epoch. All experiments were performed using PyTorch v1.11.0 on an NVIDIA Titan RTX GPU.

[0070] ACL is compared with state-of-the-art methods, including VSRN, CAAN, VSE, VSRN++, CHAN, HREM, CORA, USER, and CIMN. The ACL method of this invention is built based on the USER architecture.

[0071] Tables 1 and 2 show the comparison results between the Flickr30K and MS-COCO test sets. ACL consistently demonstrates the best overall performance.

[0072] In the Flickr30K test, ACL's RSUM is 2.3% higher than the previous best method, USER. Specifically, in image-to-text retrieval, ACL outperforms USER by 1.1%, 0.5%, and 0.2% in R@1, respectively.

[0073] The corresponding values ​​are R@5 and R@10. Similarly, in text-to-image retrieval, ACL achieves a 0.5% improvement in R@1 and R@10 compared to USER.

[0074] In the MS-COCO 50% 1K test, ACL performed best, outperforming all other methods. Specifically, compared to its strongest competitor CIMN, ACL improved RSUM by 4.0%. In image-to-text retrieval, ACL outperformed CIMN by 2.9% in R@1, 0.4% in R@5, and remained at the same level in R@10. In text-to-image retrieval, ACL improved R@1 by 1.5%.

[0075] In the MS-COCO5K benchmark, ACL performed best, outperforming all competing methods. Compared to the strongest previous method, USER, ACL achieved a 4.1% improvement in RSUM. In the image-to-text retrieval task, ACL improved upon USER by 0.1%, 1.0%, and 0.6% in R@1, R@5, and R@10, respectively. Similarly, in text-to-image retrieval, ACL improved upon USER by 0.8%, 0.8%, and 0.8% in R@1, R@5, and R@10, respectively.

[0076] These results highlight the effectiveness of ACL in progressively aligning image-text embeddings and fusing multimodal representations, thereby mitigating semantic bias and improving retrieval performance on the Flickr30K and MS-COCO datasets.

[0077] Table 1 below shows the comparison results of the Flickr 30K test set. RSUM represents the sum of R@1, R@5, and R@10 scores in the image-to-text and text-to-image retrieval tasks.

[0078]

[0079] ACL Component Analysis. To verify the effectiveness of the proposed modules, ablation experiments were conducted on the Fli-ckr30K dataset (as shown in Table 3). Model 1 uses a traditional contrastive learning method to directly align image-text pairs. Model 2 introduces a multimodal information fusion module to align image embeddings with the fused representations; Model 3 uses the same fusion module but aligns text embeddings with their corresponding fused representations. Model 4 introduces a cross-modal momentum contrast mechanism by using a momentum-updated key encoder and a dynamic queue to generate more negative samples. Model 5 simultaneously aligns image and text embeddings with their corresponding multimodal fused representations. Models 6 and 7 combine cross-modal momentum contrast with image fusion alignment and text fusion alignment, respectively. Model 8 integrates all components.

[0080] Compared to Model 1, both Model 2 and Model 3 consistently improve retrieval performance, indicating that individually aligned image or text embeddings and their fused representations enhance cross-modal retrieval capabilities. Furthermore, Model 4 consistently outperforms Model 1, demonstrating the effectiveness of cross-modal momentum comparison.

[0081] Most importantly, comparing Model 5 with Model 1 highlights the core advantages of our proposed method. Unlike Model 1, which relies on traditional contrastive learning to directly align image-text pairs, Model 5 introduces a multimodal fusion representation as a semantic anchor, aligning both image and text embeddings to this anchor. This design effectively mitigates cross-modal semantic bias, resulting in a clear and sustained performance improvement. Further incorporating cross-modal momentum contrast into Model 5 leads to Model 8, which achieves optimal overall retrieval performance. This result demonstrates that using multimodal fusion representations as semantic anchors to align image and text embeddings, combined with cross-modal momentum contrast, effectively enhances cross-modal representation learning capabilities.

[0082] Table 2: Comparison results between the MS-COCO 5K test set and the 1K test set. RSUM represents the sum of R@1, R@5, and R@10 scores in the image-to-text and text-to-image retrieval tasks. Best results are shown in bold.

[0083]

[0084] Table 3: Flickr30K ablation experiment. Image fusion alignment refers to introducing a multimodal information fusion module to align the image embedding with the fused representation; text fusion alignment refers to aligning the text embedding with its corresponding fused representation. Cross-modal momentum contrast, by employing a momentum-updated key encoder and a dynamic queue mechanism, can generate more negative samples.

[0085]

[0086] Parameter sensitivity analysis. To investigate the loss weight parameters... θThe impact on retrieval performance was analyzed using sensitivity analyses on the Flickr30K and MS-COCO datasets, following standard practices from previous studies. Figure 2 As shown.

[0087] parameter θ Control the alignment of image and text embeddings with their corresponding fused vectors. Evaluate within a reasonable range. θ On the Flickr30K dataset, retrieval performance decreased with... θ With the increase and continuous improvement, θ The performance peaks at 0.008. Beyond this value, performance no longer improves. θ Increasing the weight leads to a performance decrease. A similar trend was observed on the MS-COCO dataset. These results indicate that moderate weight reduction is necessary. θ Improve retrieval performance by promoting semantically consistent cross-modal alignment. In contrast, larger... θ Overemphasizing this alignment may disrupt the balance of the overall training objectives and lead to performance degradation, which will be investigated in future work.

[0088] To qualitatively evaluate ACL, in Figure 3 and Figure 4 The text shows a comparison with representative search results from USER, including the top 5 image-to-text results and the top 3 text-to-image results.

[0089] like Figure 3 As shown, the first image query depicts a woman in a green vest observing a drilling rig, surrounded by onlookers; the fifth-ranked title retrieved by the user mentions "binoculars," but this item does not appear in the image. Similarly, the second image query shows a black puppy standing on a food bowl, and the third and fourth-ranked titles retrieved by the user mention "television," but this item also does not appear in the image. The titles retrieved by ACL perform better in terms of accuracy and contextual relevance.

[0090] like Figure 4 As shown, the first query text describes a young female student kicking downwards to break a wooden board held by her karate instructor, while the top-ranked image retrieved by the user shows a young boy kicking, failing to capture the specified gender. The second query text mentions street signs for "Gladys" and "Detroit," but the first image retrieved by the user shows different street names, resulting in a semantic mismatch. ACL can retrieve more accurate images for text queries, with its top-ranked image accurately reflecting the query content. These observations demonstrate that ACL improves retrieval performance by effectively mitigating cross-modal semantic bias.

[0091] In summary, we propose ACL (Cross-Modal Contrast Learning), a novel image-text retrieval method designed to mitigate semantic bias in real-world data. Due to modal differences, achieving perfect semantic alignment between images and text is challenging. Unlike traditional contrastive learning methods that rely on direct, strict alignment, ACL integrates multimodal representations as shared semantic anchors and progressively calibrates image and text embeddings toward these anchors, thus adapting to approximate matching pairings. Extensive experiments on the Flickr30K and MS-COCO benchmarks demonstrate that ACL achieves a 2.3%–4.1% improvement in RSUM and effectively mitigates cross-modal semantic bias.

[0092] Example 2: This embodiment, based on embodiment 1, further defines the specific implementation method of cross-modal momentum comparison.

[0093] The capacity of the dynamic image queue and the dynamic text queue is fixed, and in this embodiment it is set to 2048 (Flickr30K) or 4096 (MS-COCO). In each training batch, the newly generated key encoder output is added to the queue, while the earliest added embedding vector is removed to ensure that the queue always contains diverse negative samples.

[0094] In contrastive learning, image embedding vectors are compared with negative samples in a dynamic text queue, and text embedding vectors are compared with negative samples in a dynamic image queue, thereby enhancing the model's discriminative ability.

[0095] Example 3: This embodiment, based on embodiment 1, further defines the construction method of fusing multimodal representations.

[0096] Image and text features are projected into a joint embedding space of dimension 1024 through fully connected layers. The concatenated features form a joint feature sequence, which is then input to a Transformer encoder. This encoder has a single-layer structure with 16 attention heads. The encoder output is aggregated into a single fusion vector using a generalized pooling operator, serving as a shared semantic anchor.

[0097] Example 4: This embodiment provides a semantically anchored guided approximate contrastive learning system, which includes: Image encoder: Used to extract image embedding vectors from image data, including a region-level feature extraction module, a global feature extraction module, a feature enhancement module, and a pooling module; Text encoder: Used to extract text embedding vectors from text data, including a word-level feature extraction module, a projection module, and a pooling module; Cross-modal interaction module: used to build fused multimodal representations as shared semantic anchors, including Transformer encoder and generalized pooling operator; The calibration module is used to progressively calibrate image embedding vectors and text embedding vectors toward shared semantic anchors in a contrastive learning framework, generating calibrated image embedding vectors and text embedding vectors. Retrieval module: Used for image-text retrieval based on calibrated image embedding vectors and text embedding vectors.

[0098] The system may also include a cross-modal momentum comparison module, which further includes a dynamic queue module and a momentum update module for maintaining a negative sample queue and updating key encoder parameters.

[0099] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0100] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantically anchored guided approximate contrastive learning method, characterized in that, include: Acquire image and text data; Image embedding vectors and text embedding vectors are extracted using an image encoder and a text encoder, respectively. By cross-modal momentum comparison, dynamic image queues and dynamic text queues are updated, and fused multimodal representations are constructed as shared semantic anchors; In the contrastive learning framework, the image embedding vector and the text embedding vector are progressively calibrated toward the shared semantic anchor to generate calibrated image embedding vectors and calibrated text embedding vectors. Image-text retrieval based on calibrated image embedding vectors and text embedding vectors.

2. The semantic anchoring-guided approximate contrastive learning method according to claim 1, characterized in that, The specific steps for cross-modal momentum comparison are as follows: A cross-modal momentum comparison mechanism is adopted to maintain a dynamic image queue and a dynamic text queue. The dynamic image queue stores historical image embedding vectors generated by the key image encoder, and the dynamic text queue stores historical text embedding vectors generated by the key text encoder. The dynamic image queue and dynamic text queue are used as negative sample sets for comparative learning. The parameters of the key image encoder are updated based on the exponential moving average of the query image encoder parameters, and the parameters of the key text encoder are updated based on the exponential moving average of the query text encoder parameters.

3. The semantic anchoring-guided approximate contrastive learning method according to claim 1, characterized in that, Specific steps for constructing a fused multimodal representation: Image region features and text lexical features are projected into a shared joint embedding space, respectively. The projected image features are concatenated with the text features to form a joint feature sequence; The joint feature sequence is input into the Transformer encoder to model the semantic interactions within and between modes; The generalized pooling operator aggregates the interacting features into a fused multimodal representation.

4. The semantic anchoring-guided approximate contrastive learning method according to claim 1, characterized in that, The specific steps for gradually calibrating the image embedding vector and text embedding vector towards the shared semantic anchor point are as follows: using an iterative optimization method, the image embedding vector and text embedding vector are gradually approximated towards the fused multimodal representation; In each iteration, the image embedding vector and the text embedding vector are updated by a calibration function so that the image embedding vector and the text embedding vector are semantically close to the shared semantic anchor point.

5. The semantic anchoring-guided approximate contrastive learning method according to claim 1, characterized in that, In the contrastive learning framework, the training objectives include: The first loss function is the hub-aware loss function, which is used to align the query image embedding and the query text embedding in the joint embedding space within a single batch, thereby suppressing the hub problem in the high-dimensional embedding space. The second loss function is used to align the query image embedding with the key text embedding in the dynamic text queue and the query text embedding with the key image embedding in the dynamic image queue in cross-modal momentum comparison. A third loss function is used to align the query image embedding with the shared semantic anchor and to align the query text embedding with the shared semantic anchor. The overall loss function is a weighted sum of the first loss function, the second loss function, and the third loss function, and the weights of the first loss function and the third loss function are controlled by hyperparameters.

6. The semantic anchoring-guided approximate contrastive learning method according to claim 1, characterized in that, The image encoder includes: The region-level feature extraction module is used to extract multiple region-level visual features of an image through a bottom-up attention mechanism; The global feature extraction module is used to extract global visual features of an image using a pre-trained visual model. The feature enhancement module is used to project the global visual features into the same feature space as the regional visual features, calculate the attention weights between the global features and each regional feature, and modulate and enhance the regional visual features based on the attention weights to generate enhanced regional features. The pooling module is used to aggregate the enhanced regional features into an image embedding using a generalized pooling operator.

7. The semantic anchoring-guided approximate contrastive learning method according to claim 1, characterized in that, The text encoder includes: The lexical-level feature extraction module is used to extract multiple lexical-level features from the text using a pre-trained language model; The projection module is used to project the word-level features into the joint embedding space through a fully connected layer; The pooling module is used to aggregate projected word-level features into text embeddings using a generalized pooling operator.

8. A semantically anchored approximate contrastive learning system, employing the semantically anchored approximate contrastive learning method described in any one of claims 1-7, characterized in that, include: An image encoder is used to extract image embedding vectors from image data; A text encoder is used to extract text embedding vectors from text data; The cross-modal interaction module is used to construct a fusion of multimodal representations as shared semantic anchors; The calibration module is used to progressively calibrate the image embedding vector and text embedding vector toward the shared semantic anchor in the contrastive learning framework, generating calibrated image embedding vector and calibrated text embedding vector. The retrieval module is used for image-text retrieval based on calibrated image embedding vectors and text embedding vectors.

9. A semantically anchored guided approximate contrastive learning system according to claim 8, characterized in that, The cross-modal interaction module includes: Transformer encoders are used to model intramodal and intermodal interactions; Generalized pooling operators are used to aggregate multimodal features into a unified fused multimodal representation. It also includes a cross-modal momentum comparison module for maintaining a dynamic queue to store negative example embedding vectors and updating key encoder parameters.

10. A semantically anchored guided approximate contrastive learning system according to claim 9, characterized in that, The cross-modal momentum comparison module includes: The dynamic queue module is used to maintain dynamic image queues and dynamic text queues, and to store historical embedding vectors generated by the key encoder; The momentum update module is used to update the parameters of the key encoder based on the parameters of the query encoder.

Images