Image-text matching method, device, equipment, storage medium and computer program product

Abstract

The application relates to the computer technical field and discloses a kind of text-image matching method, device, equipment, storage medium and computer program product, the method comprises: in response to input query text, input query text into preset text-image matching model, extract the image feature of candidate image and the text feature of query text respectively by preset text-image matching model, preset text-image matching model is the model obtained based on text-in-model contrast loss fine-grained alignment training, fine-grained alignment training based on text-in-model contrast loss is used to enhance the distinguishing ability of model to semantic similar text, determine target image matched with query text based on image feature and text feature;Since the application is pre-trained based on text-in-model contrast loss fine-grained alignment training to text-image matching model, to enhance the distinguishing ability of model to semantic similar text, so as to improve the precision of fine-grained matching in text-image matching process, and then the accuracy of text-image matching can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and computer program product for matching text and images. Background Technology

[0002] In recent years, visual-language alignment models, represented by Contrastive Language-Image Pre-training (CLIP) models, have achieved powerful zero-shot image classification and image-text retrieval capabilities through contrastive learning on large-scale image-text pairs. However, related image-text matching models perform poorly in handling fine-grained understanding tasks (such as subtle semantic differences), resulting in low accuracy in fine-grained matching. Summary of the Invention

[0003] The main objective of this application is to provide a text-image matching method, apparatus, device, storage medium, and computer program product, which aims to solve the technical problem that related text-image matching models perform poorly in handling fine-grained understanding tasks, resulting in low accuracy in fine-grained matching.

[0004] To achieve the above objectives, this application provides an image-text matching method, which includes:

[0005] In response to the input query text, the query text is input into a preset image-text matching model;

[0006] The image features of the candidate image and the text features of the query text are extracted by the preset image-text matching model. The preset image-text matching model is a model obtained by fine-grained alignment training based on intra-text contrast loss. Fine-grained alignment training based on intra-text contrast loss is used to enhance the model's ability to distinguish semantically similar texts.

[0007] Based on the image features and the text features, a target image matching the query text is determined.

[0008] Furthermore, to achieve the above objectives, this application also proposes an image-text matching device, which includes:

[0009] The text input module is used to respond to the input query text and input the query text into a preset image-text matching model;

[0010] The feature extraction module is used to extract the image features of the candidate image and the text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained by fine-grained alignment training based on text intra-modal contrast loss. Fine-grained alignment training based on text intra-modal contrast loss is used to enhance the model's ability to distinguish semantically similar texts.

[0011] The image-text matching module is used to determine the target image that matches the query text based on the image features and the text features.

[0012] In addition, to achieve the above objectives, this application also proposes a text-image matching device, which includes a memory, a processor, and a text-image matching program stored in the memory and executable on the processor. The text-image matching program is configured to implement the text-image matching method described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium storing a text-image matching program, which, when executed by a processor, implements the text-image matching method as described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a text-image matching program, and when the text-image matching program is executed by a processor, it implements the text-image matching method as described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects:

[0016] This application discloses a method for responding to input query text by inputting the query text into a preset image-text matching model. The preset image-text matching model extracts image features of candidate images and text features of the query text. The preset image-text matching model is a model obtained by fine-grained alignment training based on intra-modal text contrast loss. Fine-grained alignment training based on intra-modal text contrast loss is used to enhance the model's ability to distinguish semantically similar texts. The target image matching the query text is determined based on image features and text features. Since this application performs fine-grained alignment training on the image-text matching model based on intra-modal text contrast loss in advance to enhance the model's ability to distinguish semantically similar texts, the accuracy of fine-grained matching in the image-text matching process can be improved, thereby improving the accuracy of image-text matching. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the image-text matching method of this application;

[0020] Figure 2 This is a flowchart illustrating the second embodiment of the image-text matching method of this application;

[0021] Figure 3 This is a flowchart illustrating the third embodiment of the image-text matching method of this application;

[0022] Figure 4 This is a schematic diagram of the module structure of the image-text matching device according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the image-text matching method in the embodiments of this application.

[0024] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0026] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0027] Visual-language alignment models, such as the Contrastive Language-Image Pre-training (CLIP) model, achieve powerful zero-shot image classification and image-text retrieval capabilities through contrastive learning on large-scale image-text pairs. However, related image-text matching models perform poorly in handling fine-grained understanding tasks (such as subtle semantic differences), resulting in low accuracy in fine-grained matching.

[0028] Therefore, to overcome the above-mentioned shortcomings, this application provides a solution, which includes: responding to the input query text, inputting the query text into a preset image-text matching model, extracting image features of candidate images and text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained by fine-grained alignment training based on intra-modal text contrast loss, the fine-grained alignment training based on intra-modal text contrast loss is used to enhance the model's ability to distinguish semantically similar text, and determining the target image that matches the query text based on image features and text features; since this application pre-trains the image-text matching model with fine-grained alignment based on intra-modal text contrast loss to enhance the model's ability to distinguish semantically similar text, it can improve the accuracy of fine-grained matching in the image-text matching process, thereby improving the accuracy of image-text matching.

[0029] It should be noted that the executing entity in this embodiment can be a text and image matching device with data processing, network communication and program running functions, such as a computer, server, or other electronic devices that can achieve the same or similar functions. This embodiment does not limit this.

[0030] Based on this, embodiments of this application provide a text-image matching method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image-text matching method of this application.

[0031] In the first embodiment, the image-text matching method includes:

[0032] Step S10: In response to the input query text, input the query text into a preset image-text matching model.

[0033] It should be understood that query text can refer to text content entered by the user to obtain the target image, containing specific semantic requirements (such as object attributes, spatial relationships, etc.), and serves as the retrieval basis for the image-text matching task. Examples include "a white cat standing on a rock" and "a man in a red shirt." The pre-defined image-text matching model can refer to the FG-CLIP 2 (Fine-Grained CLIP 2) model, a pre-built visual-language alignment model. Its core feature is the introduction of Textual Intra-modal Contrastive Loss (TIC Loss) optimization, enabling fine-grained semantic matching.

[0034] In the specific implementation, the system receives the query text input by the user (supporting Chinese or English) and directly passes it into the preset FG-CLIP 2 model.

[0035] Step S20: Extract the image features of the candidate image and the text features of the query text using the preset image-text matching model. The preset image-text matching model is a model obtained by fine-grained alignment training based on intra-text contrast loss. Fine-grained alignment training based on intra-text contrast loss is used to enhance the model's ability to distinguish semantically similar texts.

[0036] Understandably, candidate images can refer to the set of all images in the model that are to be matched with the query text. These images need to have their features extracted by the model's image encoder, and then their similarity is calculated with the query text features. For example, a set of images containing "a white cat standing on a rock," "a black cat standing on a rock," and "a white cat on the grass." Image features can refer to vector-form data extracted by the FG-CLIP 2 image encoder, used to represent key information of the image (including fine-grained content such as object category, attributes, and spatial layout). Text features can refer to vector-form data extracted by the FG-CLIP 2 text encoder, used to represent the semantic information of the query text (including subtle semantics such as core nouns, attribute descriptions, and spatial relationships). After optimization with intra-model contrastive loss, it can accurately distinguish the differences between semantically similar texts. Intra-modal contrastive loss (TIC Loss) can be considered the core loss function for fine-grained alignment training. By selecting text pairs with high semantic similarity from the target dataset (such as "running puppy" and "running dog", "red apple" and "green apple"), it narrows the feature distance of positive text pairs (semantically consistent) and widens the feature distance of highly similar negative text pairs (semantically subtle differences) in the text encoding space, directly enhancing the text encoder's ability to discriminate subtle semantic differences.

[0037] In its implementation, the FG-CLIP 2 model starts dual encoders to work in parallel. The image encoder processes all candidate images one by one to extract image features containing fine-grained information such as object attributes and spatial layout. The text encoder processes the input query text to extract text features that can reflect subtle semantics (such as "white" and "standing on a stone").

[0038] Step S30: Determine the target image that matches the query text based on the image features and the text features.

[0039] It should be understood that determining the target image that matches the query text based on image features and text features can be done by calculating the similarity between the query text and each candidate image based on the image features and text features; sorting the candidate images based on the similarity; and determining the target image that matches the query text based on the sorting results.

[0040] In the specific implementation, the candidate image features and query text features extracted by the FG-CLIP 2 model are first called, and the similarity is calculated based on a preset vector similarity algorithm (such as cosine similarity). Since the model's image features already contain fine-grained information such as object attributes and spatial layout, and the text features already have the ability to distinguish semantically similar content (derived from TIC Loss training), the similarity calculation is not just comparing "whether a certain type of object is contained", but delves into detailed dimensions such as "whether the object attributes are consistent" and "whether the spatial relationships match", ensuring that the calculation results can accurately reflect the fine-grained semantic fit between the image and text. After obtaining the similarity values ​​between the query text and each candidate image, all candidate images are sorted from high to low similarity. After sorting, the results are filtered according to the actual application requirements (such as "taking the one with the highest similarity" or "taking the one that best fits the scene in the Top-3"), and finally the image ranked first (or meeting the filtering rules) is determined as the target image.

[0041] For ease of understanding, the following example is provided, but it does not limit this application. As an example, suppose in a Chinese image-text retrieval task, a user inputs the query text "a white cat standing on a rock". This text is directly input into the FG-CLIP 2 model. The model's image encoder extracts image features from candidate images (including "white cat standing on a rock", "black cat standing on a rock", "white cat on grass", and "yellow puppy on a rock"). For the image "white cat standing on a rock", the features will focus on representing fine-grained information such as "white", "cat", "rock", and "standing". When the text encoder extracts features from the query text, due to TIC Loss training, it can clearly distinguish the semantic differences between "white" and "black", and between "standing on a rock" and "on grass". The features accurately reflect the core requirements of the text. The similarity between the text features and the features of each candidate image was calculated: the image features of "white cat standing on a rock" had the highest similarity to the query text features (fine-grained information was completely matched) and was identified as the target image; "black cat standing on a rock" was excluded because the "color" feature did not match, "white cat on grass" was excluded because the "location" feature did not match, and "yellow puppy on a rock" was excluded because the "species + color" feature did not match.

[0042] As another example, suppose in an English image-text retrieval task, a user inputs the query text "a man in a red shirt". The image encoder extracts image features from candidate images (including "a man in a red shirt", "a man in a blue shirt", and "a woman in a red shirt"), focusing on information such as "male", "red", and "shirt". The text encoder, trained with TIC Loss, can distinguish the semantic differences between "red shirt" and "blue shirt" and "man" and "woman". The extracted text features accurately correspond to the requirement of "male + red shirt". After similarity calculation, the image of "a man in a red shirt" is determined as the target image because of the perfect feature match, while "a man in a blue shirt" (color mismatch) and "a woman in a red shirt" (gender mismatch) are excluded, achieving high-precision matching.

[0043] In this embodiment, the image-text matching model is pre-trained with fine-grained alignment based on intra-modal text contrast loss to enhance the model's ability to distinguish semantically similar texts, thereby improving the accuracy of fine-grained matching in the image-text matching process and thus improving the overall accuracy of image-text matching.

[0044] Reference Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the image-text matching method of this application, based on the above. Figure 1 The first embodiment shown illustrates a second embodiment of the image-text matching method of this application.

[0045] In the second embodiment, before step S10, the method further includes:

[0046] Step S01: Construct the target dataset, wherein the target dataset includes short text descriptions, long text descriptions, and corresponding sample images.

[0047] It should be understood that, in order to improve the performance of the preset image-text matching model and significantly enhance its fine-grained discrimination capability, in this embodiment, the initial image-text matching model is trained through a two-stage progressive training method of global alignment and fine-grained fine-tuning to obtain the preset image-text matching model.

[0048] The target dataset can refer to a large-scale, high-quality set of bilingual image-text pairs specifically constructed for training the image-text matching model (FG-CLIP 2), which corely contains the correspondence between short text descriptions, long text descriptions, and sample images. In specific implementations, to support fine-grained visual-language understanding in both Chinese and English scenarios, the target dataset can also be a large-scale, high-quality set of Chinese and English bilingual image-text pairs; this embodiment does not impose any restrictions on this. Short text descriptions can refer to concise semantic summaries of sample images in the target dataset, which may include core objects or themes (such as "white cat" or "red shirt man"), used to help the model quickly establish associations between images and basic semantics. Long text descriptions can refer to detailed semantic descriptions of sample images in the target dataset, which, in addition to core objects, may also include fine-grained information such as object attributes (such as color and shape), spatial layout (such as "standing on a rock" or "on the grass"), and scene background (such as "a white cat standing on a rock" or "a man wearing a red shirt standing under a tree"), providing semantic basis for the model's fine-grained learning. Sample images can refer to image data in the target dataset that corresponds one-to-one with short and long text descriptions. They need to clearly present the core objects, attributes and spatial relationships in the text description (such as a real photo of "a white cat standing on a rock") and serve as the visual input source for the model to extract image features and achieve cross-modal alignment.

[0049] In the specific implementation, a large-scale collection of raw Chinese and English image and text data is conducted, and samples containing clear attributes and spatial relationships are selected (to ensure support for fine-grained tasks). Then, a "short text description (core semantics)" and a "long text description (fine-grained semantics)" are generated for each sample image. For example, for the image "a white cat standing on a rock", the short text "white cat" and the long text "a white cat standing on a rock" are generated, forming a high-quality bilingual dataset with a one-to-one correspondence between "short text-long text-sample image".

[0050] Step S02: Perform global alignment training on the initial image-text matching model based on the target dataset to obtain the globally aligned model.

[0051] It should be understood that the initial image-text matching model can refer to a basic visual-language model that has not undergone global alignment and fine-grained alignment training. It may have the basic architecture of an image encoder and a text encoder (similar to the basic framework of CLIP), but it is not adapted to bilingual scenarios of Chinese and English, nor does it have fine-grained semantic discrimination capabilities. It serves as the initial carrier for the subsequent two-stage training. Global alignment training can refer to the first stage of training the FG-CLIP 2 model, which is used to enable the model to establish global semantic associations between images and text (including short and long texts). This covers both coarse-grained topics (such as "cat" and its corresponding image) and initially involves some fine-grained information (such as "white" and its corresponding image), laying a stable foundation for subsequent fine-grained tuning. The global alignment model can refer to the intermediate model obtained after the first stage of global alignment training. It has the ability to match the global semantics of images and text in Chinese and English bilingual scenarios (such as associating the short text "white cat" and the long text "a white cat standing on a rock" with the corresponding images). However, it is weak in distinguishing subtle semantic differences (such as "standing on a rock" and "on the grass") and needs further fine-grained training and optimization.

[0052] In the specific implementation, the initial image-text matching model (basic dual encoder architecture) is combined with the target dataset to start the first stage of training: the image encoder of the model extracts global visual features from the sample images, and the text encoder extracts global semantic features from the corresponding short text and long text respectively. Based on the global visual features and global semantic features, the initial image-text matching model is trained for global alignment to obtain the globally aligned model.

[0053] Step S03: Perform fine-grained alignment training on the globally aligned model based on the target dataset to obtain a preset image-text matching model, wherein the fine-grained alignment training includes fine-grained alignment training based on intra-text contrast loss.

[0054] It's understandable that fine-grained alignment training refers to the second stage of training the FG-CLIP 2 model. Building upon the globally aligned model, fine-grained alignment training is introduced based on the target dataset, upgrading the model from global semantic association to fine-grained semantic precision alignment. The pre-defined image-text matching model refers to the FG-CLIP 2 model obtained after global alignment training + fine-grained alignment training. It possesses fine-grained visual-language alignment capabilities in bilingual (Chinese and English) scenarios, accurately distinguishing semantically similar image-text pairs (such as "the white cat standing on the rock" and "the white cat on the grass"). It can be used for tasks such as image-text retrieval, zero-shot classification, and visual question answering.

[0055] In the specific implementation, based on the globally aligned model, the second stage of optimization is launched based on the target dataset. Highly similar text pairs in the dataset (such as "the cat standing on the rock" and "the cat sitting on the rock") are selected, and the feature distance is optimized in the text encoding space to improve the subtle semantic discrimination power of the text encoder and obtain the preset image-text matching model.

[0056] This embodiment trains the initial image-text matching model through a two-stage progressive training method of global alignment and fine-grained tuning to obtain a preset image-text matching model, thereby improving the performance of the preset image-text matching model and significantly enhancing its fine-grained discrimination capability.

[0057] Reference Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the image-text matching method of this application, based on the above. Figure 2 The second embodiment shown presents a third embodiment of the image-text matching method of this application.

[0058] In the third embodiment, step S01 includes:

[0059] Step S011: Construct a Chinese-English image-text pair dataset, which includes short Chinese-English text descriptions, long Chinese-English text descriptions, and corresponding sample images.

[0060] It should be understood that, in order to support fine-grained visual-language understanding in both Chinese and English scenarios and to provide high-quality, bilingual, consistent fine-grained training data for the image-text matching model training, this embodiment constructs a Chinese-English image-text pair dataset, and cleans and semantically aligns the text in the English image-text pair dataset to obtain the target dataset.

[0061] The Chinese-English image-text pair dataset refers to the initial dataset used to train the FG-CLIP 2 model. It contains a one-to-one correspondence between short Chinese and English text descriptions, long Chinese and English text descriptions, and sample images. It needs to cover fine-grained semantic information (such as object attributes and spatial relationships) and serves as the foundation for subsequent data cleaning and semantic alignment. This differs from traditional image-text datasets that only contain single-language or coarse-grained information. The short Chinese and English text descriptions refer to concise semantic summaries of the sample images in the dataset. These are divided into Chinese and English short texts, containing only the core object or theme of the image (e.g., Chinese "white cat" and English "white cat"), used to help the model quickly establish bilingual associations between images and basic semantics. Long text descriptions in both Chinese and English can be detailed semantic interpretations of sample images in the dataset. They are also divided into Chinese and English categories. In addition to the core object, they also include fine-grained information such as object attributes (e.g., color "snow-white", material "rough"), spatial layout (e.g., "standing on a rock"), and scene details (e.g., "ears slightly erect"), such as "A white cat standing on a rock" in Chinese and "A white cat standing on a rock" in English. This provides bilingual semantic basis for the model's fine-grained learning.

[0062] In the specific implementation, sample images with fine-grained features (such as an image of "a white cat standing on a rock") are acquired through image acquisition (e.g., screening of public datasets, targeted shooting). Next, corresponding short Chinese and English text descriptions (Chinese "white cat", English "white cat") are generated for each sample image to focus on the core object. Then, long Chinese and English text descriptions are generated to supplement fine-grained information (Chinese "a white cat standing on a rock", English "A white catstanding on a rock"). Finally, according to the structure of "short Chinese and English text - long Chinese and English text - sample image", the initial Chinese and English image-text pair dataset is formed to ensure that the text and image content of each sample are initially matched.

[0063] Step S012: Perform data cleaning on the text in the Chinese and English image-text dataset to obtain the cleaned dataset.

[0064] Understandably, data cleaning refers to the process of optimizing the quality of text (including short / long Chinese and English text) in a dataset containing Chinese and English images and text. This includes operations such as removing erroneous information, correcting format deviations, and filtering low-quality content. The goal is to eliminate data noise, ensure the semantic accuracy and format consistency of the text, and provide high-quality text materials for subsequent semantic alignment.

[0065] In the specific implementation, for the constructed Chinese-English image-text pair dataset, the text part is optimized in multiple dimensions: (1) Eliminate incorrect texts. For example, correct the Chinese text "白se猫咪" (mixing Chinese characters and pinyin) to "白色猫咪", and correct the English text "white caton stone" (missing an article) to "a white cat on a stone". Delete texts such as "一只好看的猫咪" (lacking fine-grained information) and "a cat" (too coarse in information) that cannot support fine-grained training; (2) Filter ambiguous expressions. For example, exclude "一只站在东西上的白色猫咪" (the reference of "东西" is unclear and does not specify "石头"); (3) Unify the format specifications. For example, capitalize the first letter of English texts ("White Cat" is corrected to "white cat", only the first letter of the sentence is capitalized), use full-width punctuation for Chinese texts ("一只站在石头上的白色猫咪," is corrected to "一只站在石头上的白色猫咪,"), and delete special symbols in the text (such as "#" in "一只站在石头上的白色猫咪#"). At the same time, screen the images and retain clear samples with a high degree of match with the text, and finally form the cleaned dataset.

[0066] Step S013: Perform semantic alignment on the Chinese texts and English texts in the cleaned dataset to obtain the target dataset.

[0067] It should be understood that semantic alignment can refer to the process of performing semantic consistency verification and optimization on the Chinese texts (including short / long texts) and English texts (including short / long texts) of the same sample in the cleaned dataset. Through methods such as manual review and fine-grained semantic mapping, ensure that the bilingual texts are completely consistent in details such as core semantics, attribute features, and spatial relationships, and avoid model learning confusion caused by bilingual semantic deviations. It is a key step in forming usable training data.

[0068] In a specific implementation, based on the cleaned dataset, double alignment of core semantics + fine-grained information is performed on the bilingual texts of the same sample: (1) Core semantic alignment ensures that the Chinese short text "white cat" and the English short text "white cat" both refer to the core object of "a cat with white fur", without problems such as "white" corresponding to "black" (color deviation), "cat" corresponding to "dog" (species deviation), etc.; (2) Fine-grained information alignment verifies details such as attributes and spatial relationships in the long text one by one. For example, the Chinese "snow-white fur" should correspond to the English "snow-white fur" (instead of "white fur" to avoid missing the delicate description of "snow-white"), the Chinese "standing on a gray stone" should correspond to the English "standing on a gray stone" (instead of "standing on stone", adding the correspondence of "a" to "一块" and "gray" to "灰色"), and the Chinese "ears slightly upright" should correspond to the English "slightly upright ears" (instead of "upright ears" to retain the degree information of "slightly"); through manual review combined with the assistance of a bilingual fine-grained semantic dictionary, semantic deviations are corrected (such as adjusting "a white catstanding on a rock" to "a white cat standing on a stone" to more accurately match the semantics of the Chinese "石头"), and finally a target dataset with completely consistent bilingual semantics is formed.

[0069] Furthermore, in order to improve the efficiency of semantic difference analysis and accurately capture semantic deviations in dimensions such as attributes, spatial relationships, and detailed features, step S013 includes: analyzing the semantic differences between the Chinese text and the English text in the cleaned dataset through a pre-set large language model; based on the semantic differences, performing semantic alignment on the Chinese text and the English text through the pre-set large language model to obtain a target dataset.

[0070] It can be understood that the pre-set large language model can refer to a pre-trained language model with fine-grained semantic understanding, bilingual (Chinese - English) difference recognition, and text optimization capabilities. Its core adapts to the requirements of fine-grained vision - language alignment, can accurately capture subtle semantic information such as attributes and spatial relationships in the text, and is used to support the bilingual semantic analysis and alignment of the cleaned dataset, providing high-quality data guarantee for the subsequent training of the FG-CLIP 2 model.

[0071] In a specific implementation, a single sample (including Chinese short / long texts and English short / long texts) in the cleaned dataset is input into a pre-set large language model. First, the model passes through a fine-grained semantic extraction module to parse the core semantics and detailed features of the bilingual texts respectively. For the Chinese text (such as the long text "a white cat standing on a stone"), it extracts "object: cat, attribute: white (implied 'pure white' fineness), spatial relationship: standing on (including the 'active standing' posture), carrier: stone (implied 'gray' color attribute, corresponding to the sample image)"; for the English text (such as the cleaned long text "a white cat on a stone"), it extracts "object: cat, attribute: white (without 'pure white' fineness), spatial relationship: on (only indicating 'position on', without posture), carrier: stone (without color attribute)". Subsequently, the model passes through a semantic feature comparison module, focusing on the fine-grained dimension to locate the differences between the bilingual texts: such as the difference between "standing on (including posture)" and "on (without posture)" in spatial relationship, the difference between "implied gray stone" and "stone (without color)" in attribute, and the difference between "pure white" and "white" in attribute fineness. Finally, it outputs a fine-grained semantic difference report for this sample.

[0072] Based on the above semantic difference report, the pre-set large language model automates the adjustment of the text following the principle of first preserving the integrity of fine-grained information and then taking into account language expression habits. If the fine-grained information of the Chinese text (and the corresponding sample image) is more complete (such as including "standing posture", "gray stone", "pure white hair color"), the model supplements information and corrects the expression of the English text: optimizes "a white cat on a stone" to "a snow-white cat standing on a gray stone", supplements "snow-white" (matching "pure white"), "standing" (matching the posture of "standing on"), and "gray" (matching "gray stone"); if the English text contains fine-grained information not available in the Chinese text (such as the English "a snow-white cat with slightly upright ears on a gray stone" contains "ears slightly erected", which is not mentioned in the Chinese text), then the model supplements details to the Chinese text and corrects it to "a snow-white cat with slightly erected ears standing on a gray stone". After adjustment, the model will verify the semantic consistency of the bilingual texts again to ensure there is no fine-grained deviation, and finally integrate all the aligned samples to form the target dataset.

[0073] In this embodiment, a Chinese-English image-text pair dataset is first constructed, and then the text in the English image-text pair dataset is cleaned and semantically aligned to obtain the target dataset. This enables fine-grained visual-language understanding in both Chinese and English scenarios and provides high-quality, bilingual, consistent fine-grained training data for image-text matching model training.

[0074] In the third embodiment, step S02 includes:

[0075] Step S021: Extract the global visual features of the sample image through the initial image-text matching model, and extract the global semantic features of the short text description and the long text description.

[0076] It should be understood that, in order to establish coarse-grained and partially fine-grained semantic associations between images and text in the early stages of model training, and to lay the foundation for subsequent stages, in this embodiment, global visual features of sample images are extracted through an initial image-text matching model, and global semantic features of short and long text descriptions are extracted. Contrastive learning loss is calculated based on global visual features and global semantic features, and global alignment training is performed on the initial image-text matching model according to the contrastive learning loss to obtain a globally aligned model.

[0077] Global visual features can refer to vector data extracted by the image encoder of the initial image-text matching model, representing the overall content and core elements of the sample image. This feature focuses on thematic information of the image (such as the overall combination of "cat + stone"), rather than fine-grained details such as object attributes (such as "white" or "gray") or spatial relationships (such as "standing"). Global semantic features can refer to vector data extracted by the text encoder of the initial image-text matching model, representing the overall semantics and core theme of short or long text descriptions. This feature focuses on the core semantics of the text (such as "white + cat" in the short text "white cat," and "white cat + standing on a stone" in the long text "a white cat standing on a stone"), without delving into subtle semantic differences.

[0078] In the specific implementation, sample images from the target dataset, along with corresponding short and long Chinese and English text descriptions, are input into the initial image-text matching model. The model's image encoder processes the sample images, extracting global visual features through convolution, pooling, or attention mechanisms. This process focuses on the overall content of the image, capturing global information about the core object and the basic scene (e.g., the features of the image "a white cat standing on a rock" include the overall outline and combination relationship of "cat" and "rock," but do not highlight color attributes such as "white" and "gray"). The model's text encoder processes the short and long Chinese and English texts respectively, extracting global semantic features through word embedding and attention calculation. For the short text "white cat," the features focus on the core semantics of "white + cat"; for the long text "a white cat standing on a rock," the features focus on the core scene semantics of "white cat + standing on a rock," without delving into fine-grained details.

[0079] Step S022: Calculate the contrastive learning loss based on the global visual features and the global semantic features.

[0080] It is understandable that contrastive learning loss can be considered a loss function used in global alignment training to measure the degree of semantic association between the global visual features of a sample image and the global semantic features of short / long text. Its core logic is to narrow the feature distance between matching image-text pairs (such as "white cat image" and "white cat" text) and widen the feature distance between mismatched image-text pairs (such as "white cat image" and "black dog" text), thereby establishing an optimization objective for image-text semantic association.

[0081] In the specific implementation, the global visual features of the sample image are used as the core. Positive sample pairs (semantic matching) are formed with the global semantic features of the corresponding short and long Chinese / English texts, respectively. Simultaneously, negative sample pairs (semantic mismatch) are formed with the global semantic features of other samples' short / long texts (e.g., "black dog" and "blue dog" text features). A contrastive learning loss function (e.g., InfoNCE) is used to calculate the loss value: for positive sample pairs, the closer their features are, the smaller the loss; for negative sample pairs, the farther their features are, the smaller the loss. Finally, the weighted sum of the loss values ​​of all positive and negative sample pairs is obtained to obtain the total contrastive learning loss for that sample, which serves as the error signal for model optimization.

[0082] Step S023: Perform global alignment training on the initial image-text matching model based on the contrastive learning loss to obtain the globally aligned model.

[0083] It should be understood that global alignment training refers to the process of iteratively optimizing the parameters of the image encoder and text encoder of the initial image-text matching model based on the target dataset and with contrastive learning loss as the optimization objective. The core of the training is to enable the model to learn the global semantic associations between sample images and short texts, and between sample images and long texts, rather than fine-grained alignment. This is the first stage of the two-stage training of FG-CLIP 2.

[0084] In the specific implementation, the contrastive learning loss of all samples is aggregated, and the parameters of the image encoder and text encoder of the initial image-text matching model are updated through backpropagation algorithms (such as gradient descent). The feature extraction weights of the image encoder are adjusted to more accurately capture global visual information associated with text semantics; the weights of the text encoder are adjusted to more accurately capture global semantic information associated with the image. Through multiple rounds of iterative training (each round traversing the target dataset once), the contrastive learning loss is continuously reduced until the loss value converges (or the preset number of training rounds is reached). At this point, the model can stably establish global semantic associations between sample images and bilingual short / long texts, and finally outputs the globally aligned model.

[0085] This embodiment extracts global visual features from sample images through an initial image-text matching model, and extracts global semantic features from short and long text descriptions. Based on the global visual and semantic features, it calculates the contrastive learning loss and performs global alignment training on the initial image-text matching model according to the contrastive learning loss to obtain a globally aligned model. This allows the model to establish coarse-grained and partially fine-grained semantic associations between images and text in the early stages of training, laying the foundation for subsequent stages.

[0086] In the third embodiment, step S03 includes:

[0087] Step S031: Based on the target dataset and the fine-grained learning objective, perform fine-grained alignment training on the globally aligned model to obtain the target-learned model, wherein the fine-grained learning objective includes at least one of a region-level alignment objective, a discriminative learning objective, and a cross-modal ranking objective.

[0088] It should be understood that, in order to improve the effect of fine-grained alignment training and further enhance the model's ability to distinguish semantically similar texts, in this embodiment, the globally aligned model can be trained with fine-grained alignment based on the fine-grained learning objective and the intra-text contrast loss to obtain a preset image-text matching model.

[0089] Among these objectives, region-level alignment can focus on the precise correspondence between local image regions and text fragments. This involves aligning local features between candidate bounding boxes in the image and noun phrases in the text (such as "white cat" or "gray stone"), allowing the model to capture the connection between local image details and text fragments. Discriminative learning can introduce attribute-level hard-to-bear samples (image-text pairs that are semantically similar but have subtle differences), enhancing the model's sensitivity to subtle differences in object attributes (such as color and shape) and preventing the model from confusing similar semantics. Cross-modal ranking can optimize the ranking logic of image-text pairs, enabling the model to accurately rank single text-multiple candidate images or single image-multiple candidate text pairs based on semantic matching, thus improving the practicality of the retrieval task.

[0090] In specific implementations, fine-grained learning objectives can include at least one of region-level alignment objectives, discriminative learning objectives, and cross-modal ranking objectives, or all of these objectives. If the fine-grained learning objectives are region-level alignment objectives, discriminative learning objectives, and cross-modal ranking objectives, then the globally aligned model is trained using these objectives to obtain the target-learned model.

[0091] Furthermore, to upgrade the model from global semantic association to local semantic precision matching, the fine-grained learning objective includes a region-level alignment objective. Step S031 includes: extracting image region candidate boxes from the sample images and extracting text noun phrases from the short text description and the long text description; extracting fine-grained visual features from the image region candidate boxes and extracting fine-grained semantic features from the text noun phrases using the globally aligned model; and performing region-level alignment training on the globally aligned model based on the fine-grained visual features and the fine-grained semantic features to obtain the target-learned model.

[0092] It should be understood that image region candidate boxes refer to rectangular regions containing key local information extracted from sample images using object detection algorithms (such as Faster R-CNN, YOLO, etc.). Each box corresponds to a local object or part of an object in the image (such as the head or body of a cat, or the entire area of ​​a stone), used to focus on fine-grained local features of the image, rather than global content, and is the direct source for fine-grained visual feature extraction. Text noun phrases refer to phrases representing object entities or parts of objects extracted from short or long text descriptions using text semantic parsing tools (such as part-of-speech tagging, named entity recognition). They need to precisely correspond to the local content of the image region candidate boxes; for example, extracting "white cat" from "white cat" and "gray stone" from "standing on a gray stone," serving as input for fine-grained semantic feature extraction. Fine-grained visual features refer to vector data representing detailed information of local regions extracted from image region candidate boxes by an image encoder (after parameter fine-tuning) of a globally aligned model. Unlike the "topic-level information" of global visual features, this feature focuses on local attributes (such as the "snow-white" color of a cat's fur, the "roughness" of a stone's surface, and the "erectness" of a cat's ears). Fine-grained semantic features can refer to vector data representing local semantic details extracted from noun phrases in text by a text encoder (after parameter fine-tuning) of a globally aligned model. Corresponding to the semantic needs of fine-grained visual features, for example, extracting "white (fur color) + cat (entity)" from "white cat", and extracting "gray (color) + stone (entity) + rough (surface)" from "gray stone".

[0093] In the specific implementation, object detection technology (such as Faster R-CNN) is used to process the sample images: First, global features of the image are extracted through the backbone network (such as ResNet), and then the Region Proposal Network (RPN) is used to generate initial candidate boxes that may contain key objects; then, the classifier is used to label the objects in the candidate boxes with categories (such as "cat" and "stone"), and the regressor is used to optimize the boundary positions of the candidate boxes to ensure that the selected area accurately covers the objects; finally, the local feature extraction module of the image encoder extracts fine-grained visual features from each optimized candidate box (such as extracting "white fur and erect ears" features from the "cat body" candidate box, and extracting "gray and rough surface" features from the "stone" candidate box), thus completing the extraction of local information of the image.

[0094] Perform natural language processing on the Chinese and English texts corresponding to the sample image: First, split the text through a word segmentation tool (for example, split the long Chinese text "A white cat stands on a gray stone" into "A", "white", "cat", "stands on", "gray", "stone", "on"); then identify nouns and adjectives modifying nouns through a part-of-speech tagging tool; finally, extract phrases or single nouns formed by "adjective + noun" as noun phrases (for example, extract "white cat" and "gray stone" in Chinese, and extract "white cat" and "gray stone" in English), and extract fine-grained semantic features for each noun phrase through the phrase feature extraction module of the text encoder.

[0095] Map the fine-grained visual features of the image region candidate boxes and the fine-grained semantic features of the text noun phrases to the same high-dimensional feature space (through unified processing of the feature dimensions), and calculate the similarity (such as cosine similarity) of each pair of "region candidate box - noun phrase"; set a similarity threshold (such as 0.8), pair the "region candidate box - noun phrase" with a similarity higher than the threshold, and finally establish an accurate correspondence between "image local - text local". For example, the candidate box of "cat body" is paired with "white cat" (Chinese) and "white cat" (English), and the candidate box of "stone" is paired with "gray stone" (Chinese) and "gray stone" (English).

[0096] Furthermore, in order to enable the text encoder to accurately encode attribute semantics and then联动 the image encoder to achieve accurate matching of image attribute features - text attribute semantics, the fine-grained learning objective includes a discriminative learning objective, and the step S031 includes: screening attribute-level difficult negative sample pairs from the target dataset according to the screening principle of consistent theme and different attributes; inputting the attribute-level difficult negative sample pairs into the globally aligned model, and calculating the discriminative learning loss of the attribute-level difficult negative sample pairs according to the output result of the globally aligned model; performing discriminative learning training on the globally aligned model based on the discriminative learning loss to obtain the target learned model.

[0097] Understandably, attribute-level hard-to-bear sample pairs refer to sample pairs selected from the target dataset that satisfy the same topic category but differ only in core attributes. For example, "red apple" and "green apple" (Chinese), "a red shirt" and "a blue shirt" (English). Their topics are "apple" and "shirt" respectively (same category), differing only in the core attribute of color. These sample pairs pose a significant challenge to the model due to their high topic similarity and subtle attribute differences, hence the term "attribute-level hard-to-bear sample pairs." Discriminative learning loss refers to the loss calculation method used in training attribute-level hard-to-bear sample pairs. Essentially, it quantifies the feature distance between attribute-level hard-to-bear sample pairs, aiming to widen the feature distance between sample pairs with attribute differences. This forces the model to capture subtle semantic differences at the attribute level, focusing more on the discrimination of attribute details.

[0098] In the specific implementation, sample pairs are selected based on the core principle of consistent theme and different attributes: The first step is to determine the theme category: prioritize high-frequency fine-grained themes (such as "fruit", "clothing", "animals", etc.) to ensure that the sample pairs belong to the same theme (such as both belonging to the "apple" and "shirt" categories); the second step is to identify the core attributes: for the selected theme, extract the key attribute dimensions that need to be distinguished (such as "apple" corresponding to "color" and "ripeness", and "shirt" corresponding to "color" and "style"); the third step is to construct hard-to-bear sample pairs: under the same theme, select text pairs that "only differ in target attribute and are consistent in other information" (constructed separately for Chinese and English). For example, in the Chinese scenario, select "red apple" and "green apple" (only color attribute is different), "ripe banana" and "unripe banana" (only ripeness attribute is different); in the English scenario, select "a red shirt" and "a blue shirt" (only color attribute is different), "a long skirt" and "a short skirt" (only length attribute is different). At the same time, exclude sample pairs that "do not match the theme" (such as "apple" and "pear") or "have multiple attribute differences" (such as "big red apple" and "small green apple") to ensure that the differences focus only on a single core attribute.

[0099] The filtered Chinese-English attribute-level hard negative sample pairs (such as the Chinese "red apple", "green apple", and the English "ared shirt", "a blue shirt") are respectively input into the text encoder to generate corresponding fine-grained semantic feature vectors. A discriminative learning loss (such as an improved contrastive loss) is used to calculate the feature distance (such as cosine distance) of each pair of hard negative samples. The goal is to maximize the feature distance of the sample pairs with attribute differences. For example, if the distance between the vector of the red apple and the vector of the green apple is too close (the model fails to distinguish between "red" and "green"), the loss value increases; if the distance between the vector of a red shirt and the vector of a blue shirt is too close (the model fails to distinguish between "red" and "blue"), the loss value also increases, thereby backpropagating the signal that needs to distinguish attribute differences. With the goal of minimizing the discriminative learning loss, the parameters of the text encoder (and the associated image encoder, which needs to be linked during subsequent text-image matching) are updated through backpropagation: in each iteration, the model gradually adjusts the feature encoding weights for attribute semantics (such as enhancing the dimensional differences of the features corresponding to "red" and "green") until the discrimination accuracy of the model for the attribute-level hard negative sample pairs (such as the accuracy of determining that "red apple" and "green apple" have different semantics) on the validation set tends to be stable, ensuring that the model can stably capture subtle attribute differences.

[0100] Further, to improve the fine-grained sorting accuracy and solve the problem of chaotic matching priorities, the fine-grained learning objective includes a cross-modal sorting objective. The step S031 includes: constructing at least one set of training samples with the target data set as the data source, where each set of training samples includes a core text, an image matching the core text, an image partially matching the core text, and an image not matching the core text; calculating the similarity scores of each text-image pair in the training samples, and sorting each text-image pair according to the similarity scores to obtain a similarity sorting result; constructing a sorting loss based on the similarity sorting result, and performing cross-modal sorting training on the globally aligned model based on the sorting loss to obtain the target learned model.

[0101] It is understandable that cross-modal ranking optimization refers to constructing image-text pairs with different semantic matching degrees as training samples, and using a ranking loss function to optimize the image-text similarity ranking output by the model, so that the model can rank the image-text pairs in the order of semantic matching degree: complete match → partial match → no match. Its core function is to strengthen the model's ability to judge cross-modal matching priority in fine-grained scenarios, and it needs to cover Chinese and English bilingual scenarios to provide support for high-precision ranking in subsequent image-text retrieval tasks. The ranking loss can be a loss function used to quantify the difference between the actual output image-text pair ranking and the ideal ranking (matched pairs > partially matched pairs > no match pairs). Its core objective is to backpropagate optimization signals by calculating the loss value, forcing the model to adjust the image-text similarity calculation logic so that matched pairs have the highest similarity score and the highest ranking, while no match pairs have the lowest score and the lowest ranking.

[0102] In the specific implementation, each training sample is constructed according to the "single text + three images" pattern: The first step is to select the core text: priority is given to selecting Chinese and English texts containing clear fine-grained information (attributes, spatial relationships), such as the Chinese "a white cat standing on a stone" and the English "a white cat on a stone". "stone", to ensure that the text can support the difference between "complete match / partial match"; the second step is to match three types of images: match three types of images with the selected core text: (1) matching images: select images from the dataset that are completely consistent with the fine-grained information of the text (such as the image of "a white cat standing on a gray stone"); (2) partially matching images: select images with "consistent theme and fine-grained differences" (such as the image of "a white cat standing on a green grass", which only has spatial relationship differences; or the image of "a black cat standing on a gray stone", which only has color attribute differences); (3) non-matching images: select images that are not related to the theme of the text (such as the images of "red apple" and "blue shirt"); the third step is to form multiple sets of samples: each core text corresponds to a set of "1 text + 3 images" image-text pairs, and Chinese and English respectively construct sample groups of the same size to ensure training balance in bilingual scenarios.

[0103] The text (Chinese / English) of each image-text pair is input into a text encoder, and the images are input into an image encoder to extract fine-grained semantic and visual features, respectively. The similarity score between text features and image features is calculated using the cosine similarity formula to obtain the initial score for each image-text pair (e.g., matching image score S1, partially matching image score S2, and non-matching image score S3). Pairwise Loss is used to calculate the loss value. Each pair of image-text pairs in each sample is compared. If "matching image score S1 < partially matching image score S2" or "partially matching image score S2 < non-matching image score S3", a loss penalty is triggered, and the loss value increases with the degree of score inversion. If the scores satisfy the ideal order of "S1 > S2 > S3", the loss value decreases, thus quantifying the deviation between the model's ranking result and the ideal target.

[0104] With the goal of minimizing ranking loss, the parameters of the text encoder and image encoder are updated through backpropagation. In each iteration, the model gradually adjusts the weight allocation of "fine-grained features (spatial relationships, color)," for example, enhancing the visual feature difference encoding of "stone" and "grass" and the semantic feature difference encoding of "stone" and "grass," so that the similarity score of the matched image is consistently higher than that of the partially matched image, and the score of the partially matched image is consistently higher than that of the non-matched image. The ranking accuracy on the validation set (such as the proportion of "matching pairs ranked first") tends to stabilize, thus completing the cross-modal ranking objective optimization.

[0105] Step S032: Based on the target dataset and the text intra-modal contrast loss, perform fine-grained alignment training on the target-learned model to obtain a preset image-text matching model.

[0106] In the specific implementation, based on the target learning model, highly similar text pairs (such as "the cat standing on the rock" and "the cat sitting on the rock") are selected from the target dataset. The feature distance is optimized in the text encoding space to improve the subtle semantic discrimination power of the text encoder and obtain the preset image-text matching model.

[0107] Further, to improve the fine-grained training effect based on the intra-text contrast loss, step S032 includes: screening text pairs with high semantic similarity from the target dataset, and classifying the text pairs into positive text pairs and negative text pairs; inputting the positive text pairs and the negative text pairs into the target post-learned model, and converting each piece of text into a feature vector of a fixed dimension through the target post-learned model, and all the feature vectors jointly constitute a text encoding space; within the text encoding space, calculating the feature distance for each group of positive and negative text pairs respectively; constructing an intra-text contrast loss based on the feature distance, with the optimization goal of narrowing the distance of positive text pairs and pushing away the distance of negative text pairs, and performing fine-grained alignment training on the target post-learned model based on the intra-text contrast loss to obtain a preset graphic-text matching model.

[0108] It should be understood that positive text pairs can refer to text combinations in text pairs with high semantic similarity where the core semantics are completely consistent and the fine-grained differences are extremely small (not affecting the core scene matching), such as the Chinese texts "A white cat standing on a rock" and "A white kitten standing on a rock", which are used to guide the model to perceive the feature associations with consistent semantics. Negative text pairs can refer to text combinations in text pairs with high semantic similarity where the core semantics are similar but there are key fine-grained differences (affecting the core scene matching), such as the Chinese texts "A white cat standing on a rock" and "A white cat sitting on a rock" (the difference is the action of "standing" and "sitting", which changes the spatial interaction relationship between the cat and the rock), which are used to guide the model to distinguish the feature boundaries of semantic differences.

[0109] In specific implementation, first extract all Chinese and English texts (including short / long texts) from the target dataset, and screen text combinations with consistent core semantics based on a semantic similarity algorithm (such as initial screening with cosine similarity). For example, around the core scene of "A white cat on a rock", screen out Chinese texts such as "A white cat standing on a rock", "A white kitten standing on a rock", "A white cat sitting on a rock", and the corresponding English texts; then classify the text pairs according to whether the "fine-grained differences affect the core scene": those with unchanged core scenes and extremely small differences are classified as positive text pairs (such as "A white cat standing on a rock" and "A white kitten standing on a rock"), and those with associated core scenes but key detail differences are classified as negative text pairs (such as "A white cat standing on a rock" and "A white cat sitting on a rock"), ensuring that both positive and negative text pairs focus on fine-grained dimension differences.

[0110] The filtered positive and negative text pairs (Chinese and English are processed separately) are input into the target learning model. The model's text encoder (which has been trained in the early stage and has basic semantic extraction capabilities) performs semantic parsing and encoding on each text segment. The text is segmented into initial vectors through a word embedding layer, and then the semantic relationship between words is captured by the Transformer attention mechanism. Finally, a fixed-dimensional feature vector (e.g., 512-dimensional) is output. The feature vectors of all texts together constitute the text encoding space. For example, "a white cat standing on a rock" corresponds to vector V1, "a white kitten standing on a rock" corresponds to V2, and "a white cat sitting on a rock" corresponds to V3. V1, V2, and V3 are all in this space.

[0111] In the constructed text encoding space, for each pair of positive and negative texts, the cosine distance is used to calculate the distance between feature vectors. For positive text pairs (V1 and V2), the cosine distance D_pos is calculated (initially 0.4, indicating that the semantics are close but there is still room for optimization); for negative text pairs (V1 and V3), the cosine distance D_neg is calculated (initially 0.5, indicating that the semantic differences are not sufficiently distinguished). The distance calculation results of all positive and negative text pairs provide a quantitative basis for the subsequent loss construction.

[0112] The TIC Loss function is constructed based on the calculated D_pos and D_neg. The function design follows the principle that "the smaller D_pos and the larger D_neg are, the smaller the loss value". This loss function is used as the optimization objective. The text encoder parameters of the model after objective learning are updated through the backpropagation algorithm, and the attention weights are adjusted so that the model pays more attention to fine-grained semantic features such as "stand" and "sit", "cat" and "kitten". After multiple rounds of iterative training (each round traverses all positive and negative text pairs) until the TIC Loss value converges (e.g., from the initial 0.6 to 0.2), the model can accurately distinguish fine-grained semantic differences and is finally transformed into the preset image-text matching model.

[0113] This embodiment uses fine-grained learning objectives and intra-textual contrast loss to train the globally aligned model with fine-grained alignment, thereby obtaining a preset image-text matching model. This improves the effect of fine-grained alignment training and further enhances the model's ability to distinguish semantically similar texts.

[0114] For ease of understanding, the following examples are provided, but are not intended to limit this application. As an example, in recent years, visual-language alignment models, represented by CLIP, have achieved powerful zero-shot image classification and image-text retrieval capabilities through contrastive learning on large-scale image-text pairs. However, existing models perform poorly on fine-grained understanding tasks (such as distinguishing similar object attributes, spatial layout, or subtle semantic differences). The fundamental reason is that the training data is mostly coarse-grained image-short text pairs; the models tend to learn topic-level alignment and struggle to capture the precise correspondence between image regions and text fragments. Although some work has attempted to improve this, significant limitations remain:

[0115] (1) Although methods such as FineCLIP and LongCLIP introduce region signals or long texts, they do not systematically solve the fine-grained alignment problem in Chinese and English bilingual scenarios;

[0116] (2) Existing Chinese visual-language models (such as Chinese-CLIP) mainly focus on short text retrieval and lack support for fine-grained semantics (such as attributes and spatial relationships);

[0117] (3) The lack of a unified training framework and high-quality bilingual datasets makes it difficult for the model to achieve robust fine-grained alignment in both Chinese and English scenarios.

[0118] Therefore, existing technologies have significant shortcomings in supporting bilingual (Chinese and English) communication and achieving high-precision, fine-grained visual-language alignment. A novel, unified framework is urgently needed to overcome the dual challenges of language and granularity, enabling accurate semantic matching across languages ​​and modalities.

[0119] To address the aforementioned technical challenges, this application proposes FG-CLIP 2 (Fine-Grained CLIP 2), a unified fine-grained visual-language alignment method for Chinese and English bilingual languages. This method employs a two-stage progressive training paradigm, combining large-scale, high-quality bilingual data with a novel intra-textual contrast loss to achieve accurate alignment of images and text at both the region and semantic levels.

[0120] Technical Approach: The core of FG-CLIP 2 lies in its two-stage training strategy, which gradually improves the model's ability to move from global alignment to fine-grained discrimination. The first stage learns the global semantic alignment between images and texts of varying lengths; the second stage introduces fine-grained learning objectives and designs a Textual Intra-modal Contrastive (TIC Loss) to enhance the model's ability to distinguish semantically similar texts.

[0121] The specific implementation process is as follows:

[0122] 1. Bilingual Data Construction and Preprocessing: Construct a large-scale, high-quality Chinese-English image-text pair dataset, including short and long text descriptions. Clean and align the text to ensure semantic consistency between Chinese and English, supporting fine-grained tasks (such as attribute recognition and spatial relationship understanding).

[0123] 2. First Stage: Global Alignment Training. Image and text features are extracted using an image encoder and a text encoder, respectively. For each image, its corresponding short and long text descriptions are matched simultaneously, and the contrastive learning loss is calculated. This stage aims to establish coarse-grained and partially fine-grained semantic associations between images and text in the early stages of training, laying the foundation for subsequent stages.

[0124] 3. Second Stage: Fine-grained Alignment Training. Building upon the model from the first stage, the following fine-grained learning objectives are introduced:

[0125] Region-level alignment target: Local alignment is achieved by using candidate bounding boxes of image regions with noun phrases in the text.

[0126] Discriminative learning objective: Introduce attribute-level hard negatives, such as using "red apple" and "green apple" as negative pairs, to improve the model's sensitivity to subtle semantic differences.

[0127] Cross-modal ranking objective: Optimize the ranking performance of image-text pairs and improve the fine-grained discrimination capability in retrieval tasks.

[0128] We propose a Text Intramodal Contrast Loss (TIC Loss): This loss function selects text pairs with high semantic similarity from the training set (e.g., "running puppy" and "running dog"). Within the text encoding space, it brings positive text pairs closer together and pushes away highly similar negative text pairs. This loss function directly enhances the text encoder's ability to discriminate subtle semantic differences, thereby improving the accuracy of cross-modal matching.

[0129] 4. Model Inference and Application: After training, the model can be used for tasks such as image and text retrieval, zero-shot classification, and visual question answering. Given query text, the model calculates its similarity to candidate images and returns the best match. Due to fine-grained training, the model can accurately distinguish subtle differences such as "people wearing hats" and "people not wearing hats," and performs excellently in both Chinese and English scenarios.

[0130] Example Description: In a Chinese image-text retrieval task, when a user inputs "a white cat standing on a rock," the FG-CLIP 2 model can not only match images containing both "cat" and "rock," but also accurately exclude distractors such as "black cat" or "cat on grass" through fine-grained training in the second stage and TIC loss, achieving high-precision retrieval. Similarly, in an English task, the model can effectively distinguish between "a man in a red shirt" and "a man in a blue shirt."

[0131] The FG-CLIP 2 method proposed in this application has the following significant advantages compared with related technologies:

[0132] 1. Achieve fine-grained alignment between Chinese and English: For the first time, a unified framework has been built to support fine-grained visual-language understanding in both Chinese and English scenarios, filling a gap in the research of fine-grained multimodal Chinese.

[0133] 2. Two-stage training improves performance: Through a progressive strategy of "global alignment → fine-grained tuning", the model learns more stably and the fine-grained discrimination ability is significantly enhanced.

[0134] 3. TIC Loss Enhances Text Discriminative Power: The proposed Intra-Modal Contrast Loss (TIC Loss) effectively enhances the model's ability to distinguish semantically similar texts.

[0135] It should be noted that the data collection, tag management, rule setting, and push decision-making processes involved in this application are designed to work with other technical features to solve technical problems. They do not involve or support any illegal activities. Any data processing that may violate laws and regulations (such as unauthorized collection of privacy data, generation of discriminatory tags, setting unfair rules, or pushing illegal information) is not within the scope of protection of this application's technical solution. Of course, the user data in this application will be encrypted, anonymized, or de-identified before storage to ensure user data security.

[0136] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the image-text matching method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0137] This application also provides a text-image matching device; please refer to... Figure 4 The image-text matching device includes:

[0138] Text input module 10 is used to input the query text into a preset image-text matching model in response to the input query text;

[0139] The feature extraction module 20 is used to extract the image features of the candidate image and the text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained by fine-grained alignment training based on text intra-modal contrast loss. Fine-grained alignment training based on text intra-modal contrast loss is used to enhance the model's ability to distinguish semantically similar texts.

[0140] The image-text matching module 30 is used to determine the target image that matches the query text based on the image features and the text features.

[0141] The image-text matching device provided in this application, employing the image-text matching method in the above embodiments, can solve the technical problem that related image-text matching models perform poorly in handling fine-grained understanding tasks, resulting in low fine-grained matching accuracy. Compared with the prior art, the beneficial effects of the image-text matching device provided in this application are the same as those of the image-text matching method provided in the above embodiments, and other technical features in the image-text matching device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0142] This application provides a text-image matching device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the text-image matching method in Embodiment 1 above.

[0143] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a text-image matching device suitable for implementing embodiments of this application. The text-image matching device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The illustrated image-text matching device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0144] like Figure 5As shown, the image and text matching device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the image and text matching device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the text matching device to communicate wirelessly or wiredly with other devices to exchange data. Although text matching devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0145] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0146] The image-text matching device provided in this application, employing the image-text matching method in the above embodiments, can solve the technical problem that related image-text matching models perform poorly in handling fine-grained understanding tasks, resulting in low fine-grained matching accuracy. Compared with the prior art, the beneficial effects of the image-text matching device provided in this application are the same as those of the image-text matching method provided in the above embodiments, and other technical features in this image-text matching device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0147] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0148] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0149] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the image-text matching method in the above embodiments.

[0150] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image-text matching method. This addresses the technical problem that related image-text matching models perform poorly in handling fine-grained understanding tasks, resulting in low fine-grained matching accuracy. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the image-text matching method provided in the above embodiments, and will not be repeated here.

[0151] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the image-text matching method described above.

[0152] The computer program product provided in this application can solve the technical problem that related image-text matching models perform poorly in handling fine-grained understanding tasks, resulting in low fine-grained matching accuracy. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the image-text matching method provided in the above embodiments, and will not be repeated here.

[0153] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

[0154] This application discloses A1, a method for matching images and text, the method comprising:

[0155] In response to the input query text, the query text is input into a preset image-text matching model;

[0156] The image features of the candidate image and the text features of the query text are extracted by the preset image-text matching model. The preset image-text matching model is a model obtained by fine-grained alignment training based on intra-text contrast loss. Fine-grained alignment training based on intra-text contrast loss is used to enhance the model's ability to distinguish semantically similar texts.

[0157] Based on the image features and the text features, a target image matching the query text is determined.

[0158] A2. The image-text matching method as described in A1, further comprising, before inputting the query text into the preset image-text matching model in response to the input query text:

[0159] Construct a target dataset, wherein the target dataset includes short text descriptions, long text descriptions, and corresponding sample images;

[0160] Based on the target dataset, the initial image-text matching model is trained with global alignment to obtain the globally aligned model.

[0161] Based on the target dataset, the globally aligned model is trained with fine-grained alignment to obtain a preset image-text matching model. The fine-grained alignment training includes fine-grained alignment training based on intra-text contrast loss.

[0162] A3. The image-text matching method as described in A2, wherein the step of performing fine-grained alignment training on the globally aligned model based on the target dataset to obtain a preset image-text matching model includes:

[0163] Based on the target dataset and the fine-grained learning objective, the globally aligned model is trained with fine-grained alignment to obtain the target-learned model. The fine-grained learning objective includes at least one of a region-level alignment objective, a discriminative learning objective, and a cross-modal ranking objective.

[0164] Based on the target dataset and the text intra-modal contrast loss, the target-learned model is trained with fine-grained alignment to obtain a preset image-text matching model.

[0165] A4. The image-text matching method as described in A3, wherein the fine-grained alignment training of the target-learned model based on the target dataset and intra-text contrast loss to obtain a preset image-text matching model includes:

[0166] Select text pairs with high semantic similarity from the target dataset and divide the text pairs into positive text pairs and negative text pairs;

[0167] The positive text pairs and the negative text pairs are input into the target learning model, and each text segment is converted into a fixed-dimensional feature vector through the target learning model. All feature vectors together constitute the text encoding space.

[0168] Within the text encoding space, feature distances are calculated for each pair of positive and negative text.

[0169] Based on the feature distance, an intra-text contrast loss is constructed, with the optimization objective being to bring positive text pairs closer together and push negative text pairs further apart. Based on the intra-text contrast loss, the model after learning the objective is trained with fine-grained alignment to obtain a preset image-text matching model.

[0170] A5. The image-text matching method as described in A3, wherein the fine-grained learning objective includes a region-level alignment objective, and the step of performing fine-grained alignment training on the globally aligned model based on the target dataset and the fine-grained learning objective to obtain the target-learned model includes:

[0171] Image region candidate boxes are extracted from the sample images, and text noun phrases are extracted from the short text description and the long text description;

[0172] The model after global alignment extracts fine-grained visual features from the candidate bounding boxes of the image region and fine-grained semantic features from the noun phrases in the text.

[0173] Based on the fine-grained visual features and the fine-grained semantic features, the globally aligned model is trained at the region level to obtain the target-learned model.

[0174] A6. The image-text matching method as described in A3, wherein the fine-grained learning objective includes a discriminative learning objective, and the step of performing fine-grained alignment training on the globally aligned model based on the target dataset and the fine-grained learning objective to obtain the target-learned model includes:

[0175] Based on the selection principles of thematic consistency and attribute differences, attribute-level hard negative sample pairs are selected from the target dataset;

[0176] The attribute-level hard-to-bear sample pairs are input into the globally aligned model, and the discriminative learning loss of the attribute-level hard-to-bear sample pairs is calculated based on the output of the globally aligned model.

[0177] The globally aligned model is trained using the discriminative learning loss to obtain the target-learned model.

[0178] A7. The image-text matching method as described in A3, wherein the fine-grained learning objective includes a cross-modal ranking objective, and the step of performing fine-grained alignment training on the globally aligned model based on the target dataset and the fine-grained learning objective to obtain the target-learned model includes:

[0179] At least one set of training samples is constructed using the target dataset as the data source, wherein each set of training samples includes core text, images that match the core text, images that partially match the core text, and images that do not match the core text;

[0180] Calculate the similarity score of each group of image-text pairs in the training samples, and sort the group of image-text pairs according to the similarity score to obtain the similarity ranking result;

[0181] Based on the similarity ranking results, a ranking loss is constructed, and the globally aligned model is trained for cross-modal ranking based on the ranking loss to obtain the target-learned model.

[0182] A8. The image-text matching method as described in A2, wherein the step of performing global alignment training on the initial image-text matching model based on the target dataset to obtain the globally aligned model includes:

[0183] The global visual features of the sample image are extracted using the initial image-text matching model, and the global semantic features of the short text description and the long text description are also extracted.

[0184] Calculate the contrastive learning loss based on the global visual features and the global semantic features;

[0185] The initial image-text matching model is trained for global alignment based on the contrastive learning loss to obtain the globally aligned model.

[0186] A9. The image-text matching method as described in A2, wherein constructing the target dataset includes:

[0187] Construct a dataset of Chinese and English image-text pairs, which includes short Chinese and English text descriptions, long Chinese and English text descriptions, and corresponding sample images;

[0188] Data cleaning is performed on the text in the Chinese and English image-text dataset to obtain the cleaned dataset;

[0189] Semantic alignment is performed on the Chinese and English texts in the cleaned dataset to obtain the target dataset.

[0190] A10. The image-text matching method as described in A9, wherein semantic alignment is performed on the Chinese and English texts in the cleaned dataset to obtain the target dataset, includes:

[0191] The semantic differences between Chinese and English texts in the cleaned dataset are analyzed using a pre-defined large language model.

[0192] Based on the semantic differences, the Chinese text and the English text are semantically aligned using the preset large language model to obtain the target dataset.

[0193] A11. The image-text matching method as described in any one of A1 to A10, wherein determining the target image matching the query text based on the image features and the text features includes:

[0194] The similarity between the query text and each candidate image is calculated based on the image features and the text features.

[0195] The candidate images are sorted based on the similarity, and the target image that matches the query text is determined based on the sorting results.

[0196] This application also discloses B12, a text-image matching device, the text-image matching device comprising:

[0197] The text input module is used to respond to the input query text and input the query text into a preset image-text matching model;

[0198] The feature extraction module is used to extract the image features of the candidate image and the text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained by fine-grained alignment training based on text intra-modal contrast loss. Fine-grained alignment training based on text intra-modal contrast loss is used to enhance the model's ability to distinguish semantically similar texts.

[0199] The image-text matching module is used to determine the target image that matches the query text based on the image features and the text features.

[0200] B13. The image-text matching device as described in B12, further comprising:

[0201] The model training module is used to construct a target dataset, wherein the target dataset includes short text descriptions, long text descriptions, and corresponding sample images; to perform global alignment training on an initial image-text matching model based on the target dataset to obtain a globally aligned model; and to perform fine-grained alignment training on the globally aligned model based on the target dataset to obtain a preset image-text matching model, wherein the fine-grained alignment training includes fine-grained alignment training based on intra-text contrast loss.

[0202] B14. The image-text matching device as described in B13, wherein the model training module is further configured to perform fine-grained alignment training on the globally aligned model based on the target dataset and fine-grained learning objectives to obtain a target-learned model, wherein the fine-grained learning objectives include at least one of a region-level alignment objective, a discriminative learning objective, and a cross-modal ranking objective; and to perform fine-grained alignment training on the target-learned model based on the target dataset and intra-modal text contrast loss to obtain a preset image-text matching model.

[0203] B15. In the image-text matching device described in B14, the model training module is further configured to: filter text pairs with high semantic similarity from the target dataset and divide the text pairs into positive text pairs and negative text pairs; input the positive text pairs and the negative text pairs into the target learned model, and convert each text segment into a fixed-dimensional feature vector through the target learned model, with all feature vectors jointly constituting a text encoding space; calculate the feature distance for each pair of positive and negative text pairs within the text encoding space; construct an intra-model text contrast loss based on the feature distance, with the optimization objective of narrowing the distance between positive text pairs and widening the distance between negative text pairs; perform fine-grained alignment training on the target learned model based on the intra-model text contrast loss to obtain a preset image-text matching model.

[0204] B16. The image-text matching device as described in B14, wherein the fine-grained learning objective includes a region-level alignment objective, and the model training module is further configured to extract image region candidate boxes from the sample image and extract text noun phrases from the short text description and the long text description; extract fine-grained visual features from the image region candidate boxes and extract fine-grained semantic features from the text noun phrases through the globally aligned model; and perform region-level alignment training on the globally aligned model based on the fine-grained visual features and the fine-grained semantic features to obtain the target-learned model.

[0205] B17. The image-text matching device as described in B14, wherein the fine-grained learning objective includes a discriminative learning objective, and the model training module is further configured to: select attribute-level hard-to-bear sample pairs from the target dataset according to the selection principles of topic consistency and attribute difference; input the attribute-level hard-to-bear sample pairs into the globally aligned model, and calculate the discriminative learning loss of the attribute-level hard-to-bear sample pairs based on the output of the globally aligned model; and perform discriminative learning training on the globally aligned model based on the discriminative learning loss to obtain the target-learned model.

[0206] This application also discloses C18, a text-image matching device, the text-image matching device comprising: a memory, a processor, and a text-image matching program stored in the memory and executable on the processor, wherein the text-image matching program, when executed by the processor, implements the text-image matching method as described above.

[0207] This application also discloses D19, a storage medium storing a text-image matching program, which, when executed by a processor, implements the text-image matching method as described above.

[0208] This application also discloses E20, a computer program product including a text-image matching program, which, when executed by a processor, implements the text-image matching method as described above.

Claims

1. A method for matching images and text, characterized in that, The image-text matching method includes: In response to the input query text, the query text is input into a preset image-text matching model; The image features of the candidate image and the text features of the query text are extracted by the preset image-text matching model. The preset image-text matching model is a model obtained by fine-grained alignment training based on intra-text contrast loss. Fine-grained alignment training based on intra-text contrast loss is used to enhance the model's ability to distinguish semantically similar texts. Based on the image features and the text features, a target image matching the query text is determined; Before inputting the query text into the preset image-text matching model, the process also includes: Construct a target dataset, wherein the target dataset includes short text descriptions, long text descriptions, and corresponding sample images; Based on the target dataset, the initial image-text matching model is trained with global alignment to obtain the globally aligned model. Based on the target dataset and the fine-grained learning objective, the globally aligned model is trained with fine-grained alignment to obtain the target-learned model. The fine-grained learning objective includes at least one of a region-level alignment objective, a discriminative learning objective, and a cross-modal ranking objective. Text pairs with high semantic similarity are selected from the target dataset and divided into positive text pairs and negative text pairs. Positive text pairs refer to text pairs with high semantic similarity where the core semantics are completely consistent and the fine-grained differences are minimal. Negative text pairs refer to text pairs with high semantic similarity where the core semantics are similar but there are key fine-grained differences. The positive text pairs and the negative text pairs are input into the target learning model, and each text segment is converted into a fixed-dimensional feature vector through the target learning model. All feature vectors together constitute the text encoding space. Within the text encoding space, feature distances are calculated for each pair of positive and negative texts. Based on the feature distance, an intra-text contrast loss is constructed, with the optimization objective being to bring positive text pairs closer together and push negative text pairs further apart. Based on the intra-text contrast loss, the model after learning the objective is trained with fine-grained alignment to obtain a preset image-text matching model.

2. The image-text matching method as described in claim 1, characterized in that, The fine-grained learning objective includes a region-level alignment objective. The step of training the globally aligned model with fine-grained alignment based on the target dataset and the fine-grained learning objective to obtain a target-learned model includes: Image region candidate boxes are extracted from the sample images, and text noun phrases are extracted from the short text description and the long text description; The model after global alignment extracts fine-grained visual features from the candidate bounding boxes of the image region and fine-grained semantic features from the noun phrases in the text. Based on the fine-grained visual features and the fine-grained semantic features, the globally aligned model is trained at the region level to obtain the target-learned model.

3. The image-text matching method as described in claim 1, characterized in that, The fine-grained learning objective includes a discriminative learning objective. The step of performing fine-grained alignment training on the globally aligned model based on the target dataset and the fine-grained learning objective to obtain the target-learned model includes: Based on the selection principles of thematic consistency and attribute differences, attribute-level hard negative sample pairs are selected from the target dataset; The attribute-level hard-to-bear sample pairs are input into the globally aligned model, and the discriminative learning loss of the attribute-level hard-to-bear sample pairs is calculated based on the output of the globally aligned model. The globally aligned model is trained using the discriminative learning loss to obtain the target-learned model.

4. A text-image matching device, characterized in that, The image-text matching device includes: The text input module is used to respond to the input query text and input the query text into a preset image-text matching model; The feature extraction module is used to extract the image features of the candidate image and the text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained by fine-grained alignment training based on text intra-modal contrast loss. Fine-grained alignment training based on text intra-modal contrast loss is used to enhance the model's ability to distinguish semantically similar texts. The image-text matching module is used to determine the target image that matches the query text based on the image features and the text features. The image-text matching device further includes: The model training module is used to construct a target dataset, which includes short text descriptions, long text descriptions, and corresponding sample images. Based on the target dataset, an initial image-text matching model is trained for global alignment to obtain a globally aligned model. Based on the target dataset and a fine-grained learning objective, the globally aligned model is trained for fine-grained alignment to obtain a target-learned model. The fine-grained learning objective includes at least one of a region-level alignment objective, a discriminative learning objective, and a cross-modal ranking objective. Text pairs with high semantic similarity are selected from the target dataset and divided into positive and negative text pairs. Positive text pairs refer to text pairs with high semantic similarity where the core semantics are completely identical. The text pairs are consistent and have minimal fine-grained differences. The negative text pairs refer to text pairs with high semantic similarity, sharing similar core semantics but exhibiting key fine-grained differences. The positive and negative text pairs are input into the target-learned model, which transforms each text segment into a fixed-dimensional feature vector. All feature vectors together constitute the text encoding space. Within this text encoding space, feature distances are calculated for each pair of positive and negative texts. Based on these feature distances, an intra-modal text contrast loss is constructed, with the optimization objective of narrowing the distance between positive text pairs and widening the distance between negative text pairs. Fine-grained alignment training is then performed on the target-learned model based on this intra-modal text contrast loss to obtain a pre-defined image-text matching model.

5. A text-image matching device, characterized in that, The image-text matching device includes: a memory, a processor, and an image-text matching program stored in the memory and executable on the processor. When the image-text matching program is executed by the processor, it implements the image-text matching method as described in any one of claims 1 to 3.

6. A storage medium, characterized in that, The storage medium stores a text-image matching program, which, when executed by a processor, implements the text-image matching method as described in any one of claims 1 to 3.

7. A computer program product, characterized in that, The computer program product includes a text-image matching program, which, when executed by a processor, implements the text-image matching method as described in any one of claims 1 to 3.

Images