Image-text matching method and device, equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of computers, and discloses an image-text matching method, device and equipment, a storage medium and a computer program product.The method comprises the steps that in response to an input query text, the query text is input into a preset image-text matching model, image features of the candidate image and text features of the query text are extracted through a preset image-text matching model, and the preset image-text matching model is a model obtained by performing fine-grained alignment training based on text intra-model comparison loss; the fine-grained alignment training based on the in-mold comparison loss of the text is used for enhancing the distinguishing capability of the model on the semantic similar text, and a target image matched with the query text is determined based on the image features and the text features; according to the method, the fine-grained alignment training is performed on the image-text matching model in advance based on the in-mold comparison loss of the text to enhance the distinguishing capability of the model on the semantic similar text, so that the precision of fine-grained matching in the image-text matching process can be improved, and the accuracy of image-text matching can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a text-image matching method and device, equipment, a storage medium and a computer program product. BACKGROUND

[0002] In recent years, a visual-language alignment model represented by a contrastive language-image pre-training (CLIP) model realizes strong zero-shot image classification and text-image retrieval capability by contrastive learning on a large-scale text-image pair. However, the related text-image matching model performs poorly in processing fine-grained understanding tasks (such as semantic subtle differences), thereby having the defect of low fine-grained matching precision. SUMMARY

[0003] The main purpose of the present application is to provide a text-image matching method, device, equipment, a storage medium and a computer program product, aiming at solving the technical problem that the related text-image matching model performs poorly in processing fine-grained understanding tasks, thereby having low fine-grained matching precision.

[0004] To achieve the above purpose, the present application provides a text-image matching method, which comprises the following steps: In response to an input query text, inputting the query text into a preset text-image matching model; Extracting image features of a candidate image and text features of the query text by the preset text-image matching model, wherein the preset text-image matching model is a model obtained by fine-grained alignment training based on a text-in-model contrastive loss, and the fine-grained alignment training based on the text-in-model contrastive loss is used to enhance the model's ability to distinguish semantically similar texts; Determining a target image matched with the query text based on the image features and the text features.

[0005] In addition, to achieve the above purpose, the present application also provides a text-image matching device, which comprises the following modules: A text input module, configured to input a query text into a preset text-image matching model in response to the input query text; A feature extraction module, configured to extract image features of a candidate image and text features of the query text by the preset text-image matching model, wherein the preset text-image matching model is a model obtained by fine-grained alignment training based on a text-in-model contrastive loss, and the fine-grained alignment training based on the text-in-model contrastive loss is used to enhance the model's ability to distinguish semantically similar texts; A text-image matching module, configured to determine a target image matched with the query text based on the image features and the text features.

[0006] In addition, to achieve the above object, the present application also provides a picture-text matching device, which comprises a memory, a processor, and a picture-text matching program stored in the memory and executable on the processor, and the picture-text matching program is configured to implement the picture-text matching method as described above.

[0007] In addition, to achieve the above object, the present application also provides a storage medium, which stores a picture-text matching program, and the picture-text matching program is executed by a processor to implement the picture-text matching method as described above.

[0008] In addition, to achieve the above object, the present application also provides a computer program product, which comprises a picture-text matching program, and the picture-text matching program is executed by a processor to implement the picture-text matching method as described above.

[0009] The one or more technical solutions provided by the present application have at least the following technical effects: In the present application, in response to an input query text, the query text is input into a preset picture-text matching model, and image features of a candidate image and text features of the query text are extracted by the preset picture-text matching model, wherein the preset picture-text matching model is a model obtained by fine-grained alignment training based on a text-in-model contrast loss, the fine-grained alignment training based on the text-in-model contrast loss is used to enhance the ability of the model to distinguish semantically similar texts, and a target image matching the query text is determined based on the image features and the text features. Since the picture-text matching model is pre-trained based on the text-in-model contrast loss for fine-grained alignment, the ability of the model to distinguish semantically similar texts is enhanced, thereby improving the precision of fine-grained matching in the picture-text matching process, and further improving the accuracy of picture-text matching. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0012] Figure 1 The flowchart of the first embodiment of the picture-text matching method of the present application is shown in the figure. Figure 2 The flowchart of the second embodiment of the picture-text matching method of the present application is shown in the figure. Figure 3 Figure 3 is a flowchart illustrating a third embodiment of the text-image matching method of the present application; Figure 4 Figure 4 is a schematic diagram of the module structure of the text-image matching device according to an embodiment of the present application; Figure 5 Figure 5 is a schematic diagram of the device structure of the hardware operating environment involved in the text-image matching method according to an embodiment of the present application.

[0013] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0014] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0015] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The visual-linguistic alignment model represented by the contrastive language-image pre-training (CLIP) model achieves strong zero-shot image classification and image-text retrieval capability by contrastive learning on large-scale image-text pairs. However, related image-text matching models perform poorly on fine-grained understanding tasks such as semantic subtle differences, thus having the defect of low fine-grained matching precision.

[0017] Therefore, in order to overcome the above-mentioned defects, the present application provides a solution, which comprises: in response to an input query text, inputting the query text into a preset image-text matching model, extracting image features of candidate images and text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained through fine-grained alignment training based on a text-in-model contrastive loss, the fine-grained alignment training based on the text-in-model contrastive loss is used to enhance the model's ability to distinguish between semantically similar texts, and based on the image features and the text features, a target image matching the query text is determined. Since the image-text matching model is pre-trained based on the text-in-model contrastive loss for fine-grained alignment training to enhance the model's ability to distinguish between semantically similar texts, the precision of fine-grained matching in the image-text matching process can be improved, and thus the accuracy of image-text matching can be improved.

[0018] It should be noted that the execution subject of the present embodiment can be an image-text matching device having data processing, network communication and program running functions, such as a computer, a server, etc., or other electronic devices capable of achieving the same or similar functions, and the present embodiment does not limit the same.

[0019] Based on this, the embodiment of the present application provides a method for matching images and texts, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the image-text matching method of the present application is shown.

[0020] In the first embodiment, the image-text matching method comprises: Step S10: inputting the query text into a preset image-text matching model in response to the input query text.

[0021] It should be understood that the query text can refer to the text content input by the user for obtaining the target image, containing specific semantic requirements (such as object attributes, spatial relationships, etc.), which is the retrieval basis for the image-text matching task, for example, "a white cat standing on a stone" and "a man in a red shirt". The preset image-text matching model can refer to the FG-CLIP 2 (Fine-Grained CLIP 2) model, which is a pre-constructed visual-linguistic alignment model, and the core feature is to introduce a text intra-modal contrastive loss (TIC Loss) optimization, which can realize fine-grained semantic matching.

[0022] In a specific implementation, the query text input by the user (supporting Chinese or English) is directly transmitted into the preset FG-CLIP 2 model.

[0023] Step S20: extracting image features of the candidate images and text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained based on fine-grained alignment training of a text intra-modal contrastive loss, and the fine-grained alignment training based on the text intra-modal contrastive loss is used to enhance the distinguishing ability of the model for semantically similar texts.

[0024] It can be understood that the candidate images can refer to a set of all images in the model to be matched with the query text, which need to be extracted by the image encoder of the model to calculate the similarity with the query text features, for example, a set of images containing "white cat standing on the stone", "black cat standing on the stone", and "white cat on the grass". Image features can refer to vector form data extracted by the image encoder of FG-CLIP 2 for representing key information of images, including object categories, attributes, spatial layouts, and other fine-grained content. Text features can refer to vector form data extracted by the text encoder of FG-CLIP 2 for representing semantic information of the query text, including core nouns, attribute descriptions, spatial relationships, and other subtle semantics, which can accurately distinguish the differences between semantically similar texts after optimization by the text-in-text contrast loss. The text-in-text contrast loss (TIC Loss) can refer to the core loss function of fine-grained alignment training, which selects text pairs with high semantic similarity (such as "running puppy" and "running dog", "red apple" and "green apple") from the target data set, and in the text encoding space, it shortens the feature distance of positive text pairs (semantically consistent) and pushes away the feature distance of high-similarity negative text pairs (semantically subtle differences), directly enhancing the discriminative ability of the text encoder for subtle semantic differences.

[0025] In a specific implementation, the FG-CLIP 2 model starts the dual encoder to work in parallel, the image encoder processes all candidate images one by one to extract image features containing object attributes, spatial layouts, and other fine-grained information; the text encoder processes the input query text to extract text features that can reflect subtle semantics (such as "white" and "standing on the stone").

[0026] Step S30: determining a target image matching the query text based on the image features and the text features.

[0027] It should be understood that determining a target image matching the query text based on image features and text features can be calculating the similarity between the query text and each candidate image according to the image features and the text features; sorting each candidate image based on the similarity, and determining the target image matching the query text according to the sorting result.

[0028] In a specific implementation, first, the candidate image features extracted by the FG-CLIP 2 model are called with the query text features, and calculation is performed based on a preset vector similarity algorithm (such as cosine similarity). Since the image features of the model already contain object attributes, spatial layout, and other fine-grained information, and the text features already have the ability to distinguish semantically similar content (resulting from TIC Loss training), the similarity calculation is not just a comparison of “whether to contain a certain object”, but goes deep into “whether the object attributes are consistent” “whether the spatial relationship matches” and other detailed dimensions, to ensure that the calculation result can accurately reflect the degree of fit of the fine-grained semantics of the image and the text. After obtaining the similarity values of the query text and each candidate image, all candidate images are sorted in descending order of similarity. After sorting, the results are filtered according to actual application requirements (such as “take the one with the highest similarity” or “take the one that best matches the scene among the top 3”). Finally, the image at the top of the sorting (or the one that meets the filtering rules) is determined as the target image.

[0029] For ease of understanding, the following is illustrated by way of example, but does not limit the present application. As an example, assume that in a Chinese image-text retrieval task, the user inputs the query text “a white cat standing on a rock”, and directly inputs the text into the FG-CLIP 2 model. The image encoder of the model extracts the image features of the candidate images (including “a white cat standing on a rock”, “a black cat standing on a rock”, “a white cat on a grassland”, and “a yellow dog on a rock”). For the image of “a white cat standing on a rock”, the features will highlight the fine-grained information of “white”, “cat”, “rock”, and “standing”. When the text encoder extracts the features of the query text, due to TIC Loss training, it can clearly distinguish the semantic differences between “white” and “black”, and between “standing on a rock” and “on a grassland”, and the features accurately reflect the core requirements of the text. The similarity between the text features and the features of each candidate image is calculated: the image features of “a white cat standing on a rock” have the highest similarity with the query text features (the fine-grained information is completely matched), and are determined as the target image; “a black cat standing on a rock” is excluded because the “color” feature does not match, “a white cat on a grassland” is excluded because the “position” feature does not match, and “a yellow dog on a rock” is excluded because the “species + color” feature does not match.

[0030] As another example, assuming in the English image-text retrieval task, the user inputs the query text "a man in a redshirt", the image encoder extracts the image features of the candidate images (containing "a man in a red shirt", "a man in a blue shirt", "a woman in a red shirt"), and highlights the information such as "male", "red", "shirt", etc.; the text encoder is trained by TIC Loss, which can distinguish the semantic differences between "red shirt" and "blue shirt", "man" and "woman", and the extracted text features accurately correspond to the "male + red shirt" requirement; after similarity calculation, the image features of "a man in a red shirt" are completely matched, and it is determined as the target image, "a man in a blue shirt" (color inconsistency), "a woman in a red shirt" (gender inconsistency) are excluded, and high-precision matching is achieved.

[0031] The embodiment pre-trains the image-text matching model based on the text-in-model contrast loss to enhance the model's ability to distinguish similar semantics, thereby improving the precision of fine-grained matching in the image-text matching process and improving the accuracy of image-text matching.

[0032] Reference Figure 2 , Figure 2 The flowchart of the second embodiment of the image-text matching method of the present application is shown in FIG. 2, which is based on the first embodiment shown in FIG. 1. Figure 1 The second embodiment of the image-text matching method of the present application is proposed based on the first embodiment shown in FIG. 1.

[0033] In the second embodiment, before step S10, it further includes: Step S01: constructing a target data set, wherein the target data set includes short text descriptions, long text descriptions, and corresponding sample images.

[0034] It should be understood that in order to improve the performance of the preset image-text matching model and significantly enhance the fine-grained discrimination ability of the preset image-text matching model, in the embodiment, the initial image-text matching model is trained by a two-stage progressive training method of global alignment and fine-grained fine-tuning to obtain the preset image-text matching model.

[0035] The target data set can refer to a large-scale and high-quality bilingual image-text pair set specially constructed for training the image-text matching model (FG-CLIP 2), which mainly includes the corresponding relationship of short text description-long text description-sample image. In a specific implementation, in order to support fine-grained visual-linguistic understanding in Chinese and English scenarios, the target data set can also be a large-scale and high-quality Chinese-English bilingual image-text pair set, and the embodiment is not limited thereto. The short text description can refer to a concise semantic summary of the sample image in the target data set, which can include core objects or themes (such as “white cat” and “red shirt man”), and is used to help the model establish a quick association between the image and the basic semantics. The long text description can refer to a detailed semantic description of the sample image in the target data set, which can include object attributes (such as color, shape), spatial layout (such as “standing on the stone” and “on the grass”), scene background, and other fine-grained information (such as “a white cat standing on a stone” and “a man wearing a red shirt standing under a tree”), which provides a semantic basis for fine-grained learning of the model. The sample image can refer to the image data in the target data set corresponding to the short and long text descriptions, which needs to clearly present the core objects, attributes and spatial relationships in the text description (such as a real photo of “a white cat standing on a stone”), and is the visual input source for the model to extract image features and realize cross-modal alignment.

[0036] In a specific implementation, a large-scale Chinese-English image-text original data is collected, and samples containing explicit attributes and spatial relationships are selected (to ensure support for fine-grained tasks); then, “short text description (core semantics)” and “long text description (fine-grained semantics)” are generated for each sample image, for example, for the image of “a white cat standing on a stone”, the short text “white cat” and the long text “a white cat standing on a stone” are generated, forming a high-quality bilingual data set of “short text-long text-sample image” one-to-one correspondence.

[0037] Step S02: performing global alignment training on the initial image-text matching model based on the target data set to obtain a model after global alignment.

[0038] It should be understood that the initial image-text matching model can refer to a basic visual-language model that has not undergone global alignment and fine-grained alignment training, can have a basic framework of an image encoder and a text encoder (similar to the basic framework of CLIP), but is not adapted to a Chinese-English bilingual scenario and does not have fine-grained semantic distinction capability, and is an initial carrier for subsequent two-stage training. The global alignment training can refer to the first stage of FG-CLIP 2 model training, which is used to enable the model to establish global semantic association of images and texts (including short and long texts), which covers coarse-grained topics (such as "cat" and corresponding images) and also preliminarily involves part of fine-grained information (such as "white" and corresponding images), thereby laying a stable foundation for subsequent fine-grained fine-tuning. The post-global alignment model can refer to an intermediate model obtained after the first stage of global alignment training, which has global semantic matching capability of images and texts in a Chinese-English bilingual scenario (such as being able to associate "white cat" short text, "a white cat standing on a stone" long text, and corresponding images), but has weak distinction capability for semantic subtle differences (such as "standing on a stone" and "on a grassland"), and needs further fine-grained training and optimization.

[0039] In a specific implementation, the initial image-text matching model (basic dual-encoder architecture) is combined with the target data set to start the first stage of training: the image encoder of the model extracts global visual features from the sample images, the text encoder extracts global semantic features from the corresponding short text and long text respectively, and the initial image-text matching model is trained based on the global visual features and the global semantic features to obtain the post-global alignment model.

[0040] Step S03: Fine-grained alignment training is performed on the post-global alignment model based on the target data set to obtain a preset image-text matching model, wherein the fine-grained alignment training includes fine-grained alignment training based on a text-in-model contrast loss.

[0041] It can be understood that the fine-grained alignment training can refer to the second stage of FG-CLIP 2 model training, which introduces fine-grained alignment training based on the target data set on the basis of the post-global alignment model, so as to enable the model to upgrade from global semantic association to fine-grained semantic precise alignment. The preset image-text matching model can refer to the FG-CLIP 2 model finally obtained after global alignment training + fine-grained alignment training, which has fine-grained visual-language alignment capability in a Chinese-English bilingual scenario and can precisely distinguish similar image-text pairs (such as "a white cat standing on a stone" and "a white cat on a grassland"), and can be used for image-text retrieval, zero-shot classification, visual question answering, and the like.

[0042] In a specific implementation, based on the globally aligned model, the second stage optimization is started based on the target data set, high similarity text pairs in the data set are screened (such as "cat standing on the stone" and "cat sitting on the stone"), the feature distance is optimized in the text encoding space, the subtle semantic discrimination of the text encoder is improved, and the preset image-text matching model is obtained.

[0043] The embodiment trains the initial image-text matching model by the two-stage progressive training mode of global alignment and fine-grained fine-tuning to obtain the preset image-text matching model, so as to improve the performance of the preset image-text matching model and significantly enhance the fine-grained discrimination ability of the preset image-text matching model.

[0044] Reference Figure 3 , Figure 3 FIG. 1 is a flowchart of a third embodiment of an image-text matching method according to the present application, which is based on the second embodiment shown in FIG. 1. Figure 2 The third embodiment of the image-text matching method of the present application is proposed based on the second embodiment shown in FIG. 1.

[0045] In the third embodiment, the step S01 comprises: Step S011: constructing an English and Chinese image-text pair data set, wherein the English and Chinese image-text pair data set comprises English and Chinese short text descriptions, English and Chinese long text descriptions, and corresponding sample images.

[0046] It should be understood that in order to support fine-grained visual-linguistic understanding in Chinese and English scenarios, high-quality, bilingual consistent fine-grained training data is provided for image-text matching model training. In the embodiment, an English and Chinese image-text pair data set is constructed, and the texts in the English image-text pair data set are cleaned and semantically aligned respectively to obtain a target data set.

[0047] The Chinese-English image-text pair data set can refer to an initial data set constructed for training the FG-CLIP 2 model, containing a one-to-one correspondence relationship of Chinese-English short text description-Chinese-English long text description-sample image, and needs to cover fine-grained semantic information (such as object attribute, spatial relationship), which is the basis for subsequent data cleaning and semantic alignment, and is different from the traditional image-text data set containing only single language or coarse-grained information. The Chinese-English short text description can refer to a concise semantic summary of the sample image in the data set, which is divided into Chinese short text and English short text, and only contains the core object or theme of the image (such as Chinese "white cat" and English "white cat"), which is used to help the model quickly establish a bilingual association between the image and the basic semantics. The Chinese-English long text description can be a detailed semantic explanation of the sample image in the data set, which is also divided into Chinese and English, and in addition to the core object, it also contains fine-grained information such as object attribute (such as color "snow white", material "rough"), spatial layout (such as "standing on a rock"), scene details (such as "ears slightly raised"), etc. (such as Chinese "a white cat standing on a rock" and English "A white cat standing on a rock"), which provides bilingual semantic basis for fine-grained learning of the model.

[0048] In a specific implementation, sample images containing fine-grained features (such as the image of "a white cat standing on a rock") are obtained through image acquisition (such as public data set screening, directional shooting); then corresponding Chinese-English short text descriptions (Chinese "white cat" and English "white cat") are generated for each sample image, focusing on the core object; then Chinese-English long text descriptions are generated, supplementing fine-grained information (Chinese "a white cat standing on a rock" and English "A white cat standing on a rock"); finally, the initial Chinese-English image-text pair data set is formed by combining the structure of "Chinese-English short text-Chinese-English long text-sample image", ensuring that the text and image content of each sample are preliminarily matched.

[0049] Step S012: Data cleaning is performed on the text in the Chinese-English image-text pair data set to obtain a cleaned data set.

[0050] It can be understood that data cleaning can refer to a process of optimizing the quality of the text (containing Chinese and English short / long text) in the Chinese-English image-text pair data set, including operations such as removing incorrect information, correcting format deviations, and filtering low-quality content, the purpose of which is to eliminate data noise and ensure that the text semantics are accurate and the format is uniform, providing high-quality text materials for subsequent semantic alignment.

[0051] In a specific implementation, for the constructed Chinese-English image-text pair dataset, the text part is optimized in multiple dimensions: (1) incorrect texts are removed, such as correcting Chinese "white se cat" (mixed pinyin) to "white cat", English "white cat on stone" (missing article) to "a white cat on a stone", deleting texts such as "a cute cat" (no fine-grained information) and "a cat" (information is too coarse) which cannot support fine-grained training; (2) ambiguous expressions are filtered out, such as excluding "a white cat standing on something" (the meaning of "something" is not clear, "stone" is not specified); (3) format specifications are unified, such as capitalizing the first letter of English texts ("White Cat" is corrected to "white cat", only the first letter is capitalized), using full-width punctuation in Chinese texts ("a white cat standing on a stone," is corrected to "a white cat standing on a stone,"), and deleting special symbols in texts (such as "#" in "a white cat standing on a stone#"); at the same time, the images are screened, and clear samples with high matching degree with the texts are retained, and finally the cleaned dataset is formed.

[0052] Step S013: performing semantic alignment on the Chinese text and English text in the cleaned dataset to obtain a target dataset.

[0053] It should be understood that semantic alignment can refer to the process of checking and optimizing the semantic consistency of the Chinese text (including short / long text) and English text (including short / long text) of the same sample in the cleaned dataset. Through artificial review and fine-grained semantic mapping, it is ensured that the bilingual texts are completely consistent in core semantics, attribute characteristics, spatial relationships and other details, avoiding model learning confusion caused by bilingual semantic deviation, which is a key step to form usable training data.

[0054] In a specific implementation, based on the cleaned data set, the bilingual text of the same sample is aligned for core semantics + fine-grained information: (1) Core semantic alignment ensures that the Chinese short text "white cat" and the English short text "white cat" both refer to "white cat" as the core object, and there are no "white" corresponding to "black" (color deviation), "cat" corresponding to "dog" (species deviation), etc. (2) Fine-grained information alignment checks the attributes, spatial relationships, etc. in the long text one by one, such as Chinese "snow-white fur" corresponding to English "snow-white fur" (not "white fur", avoiding missing the delicate description of "snow-white"), Chinese "standing on a gray stone" corresponding to English "standing on a gray stone" (not "standing on stone", supplementing "a" corresponding to "a", "gray" corresponding to "gray"), Chinese "slightly upright ears" corresponding to English "slightly upright ears" (not "upright ears", preserving the degree information of "slightly"); Through manual review combined with bilingual fine-grained semantic dictionary assistance, semantic deviation is corrected (such as adjusting "a white cat standing on a rock" to "a white cat standing on a stone", which is more accurate in semantic matching with Chinese "stone"), and finally forming a target data set with completely consistent bilingual semantics.

[0055] Further, in order to improve the efficiency of semantic difference analysis and accurately capture semantic deviations in dimensions such as attributes, spatial relationships, and detailed features, the step S013 includes: analyzing the semantic differences between the Chinese text and the English text in the cleaned data set through a pre-set large language model; based on the semantic differences, the pre-set large language model is used to perform semantic alignment on the Chinese text and the English text to obtain a target data set.

[0056] It can be understood that the pre-set large language model can refer to a pre-trained language model with fine-grained semantic understanding, bilingual (Chinese-English) difference recognition, and text optimization capabilities. Its core adapts to the needs of fine-grained visual-linguistic alignment, and can accurately capture fine semantic information such as attributes and spatial relationships in the text, supporting bilingual semantic analysis and alignment of the cleaned data set, and providing high-quality data guarantee for subsequent FG-CLIP 2 model training.

[0057] In a specific implementation, a single sample (containing Chinese short / long text, English short / long text) in the cleaned dataset is input into the preset large language model. The model first passes through the fine-grained semantic extraction module to analyze the core semantics and detailed features of the bilingual text respectively. For Chinese text (such as long text "a white cat standing on a stone"), the model extracts "object: cat, attribute: white (implied'snow-white' delicacy), spatial relationship: standing (including 'actively standing' posture), carrier: stone (implied 'gray' color attribute, corresponding to the sample image)". For English text (such as the cleaned long text "a white cat on a stone"), the model extracts "object: cat, attribute: white (no'snow-white' delicacy), spatial relationship: on (only indicates 'position on', no posture), carrier: stone (no color attribute)". Subsequently, the model passes through the semantic feature comparison module, focuses on the fine-grained dimension, and locates the differences between the bilingual texts: such as the difference between "standing (including posture)" and "on (no posture)" in spatial relationship, the difference between "implied gray stone" and "stone (no color)" in attribute, and the difference between "snow-white" and "white" in attribute delicacy. Finally, the model outputs a fine-grained semantic difference report for the sample.

[0058] The preset large language model, based on the above semantic difference report, follows the principle of prioritizing the integrity of fine-grained information and considering language expression habits to automatically adjust the text. If the fine-grained information of the Chinese text (and the corresponding sample image) is more complete (such as including "standing posture", "gray stone", and "snow-white fur color"), the model supplements and corrects the English text: "a white cat on a stone" is optimized to "a snow-white cat standing on a gray stone", supplementing "snow-white" (matching "snow-white"), "standing" (matching the posture of "standing"), and "gray" (matching "gray stone"). If the English text contains fine-grained information that is not available in Chinese (such as English "a snow-white cat with slightly upright ears on a gray stone" containing "ears slightly upright", which is not mentioned in Chinese), the model supplements the details of the Chinese text and corrects it to "a snow-white cat with slightly upright ears standing on a gray stone". After adjustment, the model will again verify the semantic consistency of the bilingual text to ensure no fine-grained deviation, and finally integrate all aligned samples to form the target dataset.

[0059] In the embodiment, first, a Chinese-English image-text pair dataset is constructed, then the text in the English image-text pair dataset is cleaned and semantically aligned respectively to obtain a target dataset, so as to support fine-grained visual-linguistic understanding in Chinese and English scenarios, and to provide high-quality, bilingual consistent fine-grained training data for image-text matching model training.

[0060] In the third embodiment, the step S02 comprises: Step S021: extracting global visual features of the sample image by the initial image-text matching model, and extracting global semantic features of the short text description and the long text description.

[0061] It should be understood that, in order to enable the model to establish coarse-grained and partially fine-grained semantic association between images and texts at the initial stage of training, and to lay a foundation for subsequent stages, in the embodiment, global visual features of the sample image are extracted by the initial image-text matching model, and global semantic features of the short text description and the long text description are extracted, a contrast learning loss is calculated based on the global visual features and the global semantic features, and the initial image-text matching model is globally aligned and trained according to the contrast learning loss to obtain a globally aligned model.

[0062] The global visual features can be vector data representing overall content and core elements of the sample image extracted by an image encoder of the initial image-text matching model. The features focus on theme-level information (such as the overall combination of “cat + stone”) of the image, rather than object attributes (such as “white” and “gray”) and spatial relationships (such as “standing”) and other fine-grained details. The global semantic features can be vector data representing overall semantics and core themes of the short text description or the long text description extracted by a text encoder of the initial image-text matching model. The features focus on core semantics (such as “white + cat” of the short text “white cat” and “white cat + standing on the stone” of the long text “a white cat standing on the stone”) of the text, without distinguishing subtle semantic differences.

[0063] In a specific implementation, the sample image in the target data set, the corresponding Chinese and English short text description, and the Chinese and English long text description are input into an initial image-text matching model. The image encoder of the model processes the sample image, extracts global visual features of the image through convolution, pooling or attention mechanism, the process focuses on the overall content of the image, and captures global information of core objects and basic scenes (for example, the features of the "white cat standing on the stone" image will include the overall outline and combination relationship of "cat" and "stone", but will not highlight the color attributes such as "white" and "gray"); the text encoder of the model processes the Chinese and English short text and the Chinese and English long text respectively, extracts global semantic features through word embedding and attention calculation, and the features of the short text "white cat" focus on the core semantics of "white cat"; the features of the long text "a white cat standing on the stone" focus on the core scene semantics of "white cat standing on the stone", and neither of them goes into fine-grained details.

[0064] Step S022: calculating a contrastive learning loss based on the global visual features and the global semantic features.

[0065] It can be understood that the contrastive learning loss can refer to a loss function used to measure the degree of semantic association between the global visual features of the sample image and the global semantic features of the short text and the long text in global alignment training. The core logic is to pull the feature distance of the matched image-text pair (such as "white cat image" and "white cat" text) and push the feature distance of the unmatched image-text pair (such as "white cat image" and "black dog" text), so as to establish an optimization goal of image-text semantic association.

[0066] In a specific implementation, taking the global visual features of the sample image as the core, the global semantic features of the Chinese and English short text corresponding to the sample and the global semantic features of the Chinese and English long text form a positive sample pair (semantic matching), and at the same time, the global semantic features of the short text and the long text of other samples (such as "black dog" and "blue dog" text features) form a negative sample pair (semantic mismatch). A contrastive learning loss function (such as InfoNCE) is used to calculate the loss value: for the positive sample pair, the function is designed to make the feature distance closer, and the loss smaller; for the negative sample pair, the feature distance is made farther, and the loss smaller. Finally, the loss values of all positive and negative sample pairs are weighted and summed to obtain the total contrastive learning loss of the sample, which is used as an error signal for model optimization.

[0067] Step S023: globally aligning and training the initial image-text matching model according to the contrastive learning loss to obtain a globally aligned model.

[0068] It should be understood that the global alignment training can refer to a process of iteratively optimizing the image encoder and text encoder parameters of the initial image-text matching model based on the target dataset, with the contrastive learning loss as the optimization objective. The training core is to let the model learn the global semantic association of the sample image-short text and sample image-long text, rather than fine-grained alignment, which is the first stage of the two-stage training of FG-CLIP 2.

[0069] In a specific implementation, the contrastive learning loss of all samples is summarized, and the image encoder and text encoder parameters of the initial image-text matching model are updated through a back propagation algorithm (such as gradient descent) to adjust the feature extraction weights of the image encoder to more accurately capture the global visual information associated with the text semantics, and to adjust the weights of the text encoder to more accurately capture the global semantic information associated with the image. Through multiple rounds of iterative training (each round traverses the target dataset once), the contrastive learning loss is continuously reduced until the loss value converges (or the preset training rounds are reached), at which point the model has stably established the global semantic association of the sample image-bilingual short / long text, and finally outputs the globally aligned model.

[0070] The embodiment extracts the global visual features of the sample image through the initial image-text matching model, extracts the global semantic features of the short text description and the long text description, calculates the contrastive learning loss based on the global visual features and the global semantic features, and performs global alignment training on the initial image-text matching model according to the contrastive learning loss to obtain the globally aligned model, so that the model can establish coarse-grained and partially fine-grained semantic association between the image and the text in the early stage of training, laying a foundation for the subsequent stage.

[0071] In the third embodiment, the step S03 comprises: Step S031: performing fine-grained alignment training on the globally aligned model based on the target dataset and the fine-grained learning target to obtain a target learned model, wherein the fine-grained learning target comprises at least one of a region-level alignment target, a discriminative learning target, and a cross-modal ranking target.

[0072] It should be understood that in order to improve the effect of fine-grained alignment training and further improve the model's ability to distinguish between semantically similar texts, in the embodiment, the globally aligned model can be fine-grained alignment trained based on the fine-grained learning target and the text-in-model contrastive loss to obtain a preset image-text matching model.

[0073] The region-level alignment target can refer to accurate correspondence between a local region of a focus image and a text segment. Local feature alignment is performed between an image region candidate box and a noun phrase (such as "white cat" or "gray stone") in the text, so that the model can capture the association between the local details of the image and the text segment. The discriminative learning target can refer to the introduction of attribute-level hard negative samples (image-text pairs with similar semantics but slight differences), which can strengthen the sensitivity of the model to slight differences in object attributes (such as color and shape) and avoid confusion of similar semantics. The cross-modal ranking target can refer to optimization of the ranking logic of image-text pairs, so that the model can accurately rank a single text and multiple candidate images or a single image and multiple candidate texts according to the semantic matching degree, thereby improving the practicability of the retrieval task.

[0074] In a specific implementation, the fine-grained learning target can include at least one of the region-level alignment target, the discriminative learning target, and the cross-modal ranking target, or all of the region-level alignment target, the discriminative learning target, and the cross-modal ranking target. If the fine-grained learning target is the region-level alignment target, the discriminative learning target, and the cross-modal ranking target, the global alignment after model is fine-grained alignment trained based on the region-level alignment target, the discriminative learning target, and the cross-modal ranking target, and the target learning after model is obtained.

[0075] Further, in order to upgrade the model from global semantic association to local semantic accurate matching, the fine-grained learning target includes a region-level alignment target. The step S031 includes: extracting an image region candidate box from the sample image, and extracting a text noun phrase from the short text description and the long text description; extracting fine-grained visual features from the image region candidate box and extracting fine-grained semantic features from the text noun phrase by using the global alignment after model; and performing region-level alignment training on the global alignment after model based on the fine-grained visual features and the fine-grained semantic features, to obtain the target learning after model.

[0076] It should be understood that the image region candidate box can refer to a rectangular region box containing local key information extracted from a sample image by a target detection algorithm (such as Faster R-CNN, YOLO, etc.). Each box corresponds to a local object or object part in the image (such as the head and body of a cat, the overall region of a stone), which is used to focus on the fine-grained local features of the image rather than the global content, and is a direct source of fine-grained visual feature extraction. The text noun phrase can refer to a phrase representing an object entity or object part extracted from a short text description or a long text description by a text semantic analysis tool (such as part-of-speech tagging, named entity recognition). It needs to be accurately corresponding to the local content of the image region candidate box, such as extracting “white cat” from “white cat” and extracting “gray stone” from “standing on the gray stone”, which is the input of fine-grained semantic feature extraction. Fine-grained visual features can refer to vector data representing local region detail information extracted from image region candidate boxes by image encoders of the post-global alignment model (after parameter fine-tuning). Unlike the “topic-level information” of global visual features, this feature focuses on local attributes (such as “snow white” of cat hair, “rough” of stone surface, and “upright” of cat ears). Fine-grained semantic features can refer to vector data representing local semantic details extracted from text noun phrases by text encoders of the post-global alignment model (after parameter fine-tuning). Corresponding to the semantic requirements of fine-grained visual features, such as extracting “white (hair color) + cat (entity)” from “white cat” and extracting “gray (color) + stone (entity) + rough (surface)” from “gray stone”.

[0077] In a specific implementation, a target detection technology (such as Faster R-CNN) is used to process sample images: first, the global features of the image are extracted by a backbone network (such as ResNet), and then the region proposal network (RPN) is used to generate initial candidate boxes that may contain key objects; then, the classifier is used to label the objects in the candidate boxes by class (such as “cat” and “stone”), and the regressor is used to optimize the boundary position of the candidate box to ensure that the box selects the region accurately covering the object; finally, the local feature extraction module of the image encoder is used to extract fine-grained visual features (such as “white hair, upright ears” features from the “cat body” candidate box and “gray, rough surface” features from the “stone” candidate box) from each optimized candidate box, completing the extraction of image local information.

[0078] The Chinese and English texts corresponding to the sample image are subjected to natural language processing: first, the text is split by a word segmentation tool (for example, the Chinese long text "a white cat stands on a gray stone" is split into "a", "white", "cat", "stands on", "gray", "stone", and "above"); then the part-of-speech tagging tool is used to identify nouns and adjectives that modify nouns; finally, the phrases or single nouns composed of "adjective + noun" are extracted as noun phrases (for example, "white cat" and "gray stone" are extracted in Chinese, and "white cat" and "gray stone" are extracted in English), and the phrase feature extraction module of the text encoder is used to extract fine-grained semantic features for each noun phrase.

[0079] The fine-grained visual features of the image region candidate box and the fine-grained semantic features of the text noun phrase are mapped to the same high-dimensional feature space (processed by feature dimension unification), and the similarity (such as cosine similarity) of each pair of "region candidate box-noun phrase" is calculated; a similarity threshold (such as 0.8) is set, and the "region candidate box-noun phrase" with a similarity higher than the threshold is paired, and finally the precise correspondence between "image local-text local" is established, for example, the "cat body" candidate box is paired with "white cat" (Chinese) and "white cat" (English), and the "stone" candidate box is paired with "gray stone" (Chinese) and "gray stone" (English).

[0080] Further, in order to enable the text encoder to accurately encode attribute semantics, and then link the image encoder to realize precise matching of image attribute features-text attribute semantics, the fine-grained learning target includes a discriminative learning target, and the step S031 includes: selecting attribute-level hard-negative sample pairs from the target data set according to the filtering principle of consistent theme and different attributes; inputting the attribute-level hard-negative sample pairs into the global alignment model, and calculating the discriminative learning loss of the attribute-level hard-negative sample pairs according to the output result of the global alignment model; based on the discriminative learning loss, the global alignment model is subjected to discriminative learning training to obtain a target learned model.

[0081] It can be understood that the attribute-level difficult negative sample pair can refer to a sample pair filtered from the target data set, which meets the theme category consistency and only has differences in the core attributes. For example, "red apple" and "green apple" (Chinese), "a red shirt" and "a blue shirt" (English), the themes are "apple" and "shirt" respectively (category consistent), and the difference is only in the color, which is the core attribute. Such sample pairs are difficult to distinguish because of high theme similarity and subtle attribute differences, so they are called attribute-level difficult negative sample pairs. The discriminative learning loss can refer to the loss calculation method used in attribute-level difficult negative sample training. The essence is to quantify the feature distance of the attribute-level difficult negative sample pair, to optimize the feature distance of the sample pair with attribute differences, and force the model to capture the subtle semantic differences in the attribute level and focus more on the attribute details of the discrimination.

[0082] In a specific implementation, the sample pairs are filtered according to the core principles of theme consistency and attribute difference: first, determine the theme category: prefer high-frequency fine-grained themes (such as "fruit", "clothing", "animal", etc.), and ensure that the sample pairs belong to the same theme (such as "apple" and "shirt"); second, lock the core attribute: for the selected theme, extract the key attribute dimension that needs to be distinguished (such as "color" and "maturity" for "apple", and "color" and "style" for "shirt"); third, construct difficult negative sample pairs: under the same theme, filter the text pairs that are only different in the target attribute and consistent in other information (Chinese and English are constructed respectively), for example, in the Chinese scenario, filter "red apple" and "green apple" (only color attribute is different), "ripe banana" and "unripe banana" (only maturity attribute is different); in the English scenario, filter "a red shirt" and "a blue shirt" (only color attribute is different), "a long skirt" and "a short skirt" (only length attribute is different), while excluding sample pairs with "different themes" (such as "apple" and "pear") or "multiple attribute differences" (such as "red large apple" and "green small apple"), to ensure that the difference focuses only on a single core attribute.

[0083] The screened attribute-level hard-negative sample pairs (such as Chinese "red apple" and "green apple", and English "a red shirt" and "a blue shirt") are respectively input into the text encoder to generate corresponding fine-grained semantic feature vectors. A discriminative learning loss (such as an improved contrastive loss) is used to calculate the feature distance (such as the cosine distance) of each pair of hard-negative samples. The goal is to maximize the feature distance of sample pairs with attribute differences. For example, if the distance between the vector of a red apple and the vector of a green apple is too close (the model does not distinguish between "red" and "green"), the loss value increases. If the distance between the vector of a red shirt and the vector of a blue shirt is too close (the model does not distinguish between "red" and "blue"), the loss value also increases. In this way, the signal that needs to distinguish attribute differences is transmitted in reverse. To minimize the discriminative learning loss, the parameters of the text encoder (and the associated image encoder, which needs to be linked during subsequent image-text matching) are updated through backpropagation: in each iteration, the model gradually adjusts the feature encoding weights of the attribute semantics (such as enhancing the difference in feature dimensions corresponding to "red" and "green"), until the model's accuracy in distinguishing attribute-level hard-negative sample pairs (such as the accuracy of distinguishing "red apple" and "green apple" as different semantics) on the validation set tends to be stable, ensuring that the model can stably capture subtle attribute differences.

[0084] Further, in order to improve the fine-grained sorting accuracy and solve the problem of chaotic matching priority, the fine-grained learning target includes a cross-modal sorting target. The step S031 includes: constructing at least one group of training samples using the target data set as a data source, wherein each group of training samples includes a core text, an image matched with the core text, an image partially matched with the core text, and an image not matched with the core text; calculating the similarity scores of each group of image-text pairs in the training samples, and sorting each group of image-text pairs according to the similarity scores to obtain a similarity sorting result; constructing a sorting loss according to the similarity sorting result, and performing cross-modal sorting training on the globally aligned model based on the sorting loss to obtain a target learned model.

[0085] It can be understood that the cross-modal ranking target optimization can refer to the process of constructing different semantic matching degrees of image-text pairs as training samples, optimizing the image-text similarity ranking output by the model with the help of the ranking loss function, so that the model can sort the image-text pairs in the order of complete matching → partial matching → no matching semantic matching degree. Its core role is to strengthen the model's cross-modal matching priority judgment ability in the fine-grained scene, and needs to cover the English-Chinese bilingual scene, providing support for high-precision sorting for subsequent image-text retrieval tasks. The ranking loss can refer to the loss function used to quantify the difference between the actual output of the model and the ideal ranking (matching pair > partial matching pair > no matching pair). Its core goal is to calculate the loss value and pass back the optimization signal, forcing the model to adjust the image-text similarity calculation logic, so that the matching pair has the highest similarity score and the highest ranking, and the no matching pair has the lowest score and the lowest ranking.

[0086] In a specific implementation, each group of training samples is constructed in the mode of "single text + three images": first, select the core text: preferentially select Chinese and English texts containing explicit fine-grained information (attributes, spatial relationships), such as Chinese "white cat standing on stone" and English "a white cat on a stone", to ensure that the text can support the difference between "complete matching / partial matching"; second, match three types of images: match three types of images for the selected core text: (1) matching image: select images from the dataset that are completely consistent with the fine-grained information of the text (such as images of "white cat standing on gray stone"); (2) partial matching image: select images with "consistent theme and fine-grained difference" (such as images of "white cat standing on green grass", only spatial relationship difference; or images of "black cat standing on gray stone", only color attribute difference); (3) no matching image: select images that are not related to the theme of the text (such as images of "red apple" and "blue shirt"); third, form multiple groups of samples: each core text corresponds to a group of "1 text + 3 image" image-text pair sets, and Chinese and English samples of the same size are constructed to ensure balanced training in the bilingual scene.

[0087] The text (Chinese / English) in each text-image pair is input into the text encoder and the image is input into the image encoder to extract fine-grained semantic features and visual features, respectively. The similarity score of "text features-image features" is calculated by the cosine similarity formula to obtain the initial score of each text-image pair (e.g., matching image score S1, partially matching image score S2, and non-matching image score S3). Pairwise Loss is used to calculate the loss value, and the text-image pairs in each group of samples are compared pairwise. If "matching image score S1 < partially matching image score S2" or "partially matching image score S2 < non-matching image score S3", the loss penalty is triggered, and the loss value increases with the increase of the inverse order of the score; if the score satisfies the ideal order "S1 > S2 > S3", the loss value decreases, thereby quantifying the deviation of the model ranking result from the ideal target.

[0088] To minimize the ranking loss, the parameters of the text encoder and the image encoder are updated through backpropagation: in each iteration, the model gradually adjusts the weight distribution of "fine-grained features (spatial relationship, color)", for example, enhances the visual feature difference coding of "stone" and "grass" and the semantic feature difference coding of "stone" and "grass", so that the similarity score of the matching image is continuously higher than that of the partially matching image, and the score of the partially matching image is continuously higher than that of the non-matching image. When the ranking accuracy (e.g., the proportion of "matching pairs ranked first") on the validation set tends to be stable, the cross-modal ranking optimization target is completed.

[0089] Step S032: Fine-grained alignment training of the target learned model based on the target data set and the text-in-model contrastive loss to obtain a preset text-image matching model.

[0090] In a specific implementation, based on the target learned model, high-similarity text pairs (e.g., "cat standing on stone" and "cat sitting on stone") in the data set are selected based on the target data set, the feature distance is optimized in the text encoding space, the fine-grained semantic discrimination of the text encoder is improved, and a preset text-image matching model is obtained.

[0091] Further, in order to improve the fine-grained training effect based on the text intra-model contrast loss, the step S032 comprises: screening text pairs with high semantic similarity from the target data set, and dividing the text pairs into positive text pairs and negative text pairs; inputting the positive text pairs and the negative text pairs into the target learned model, and converting each text into a fixed-dimensional feature vector by the target learned model, and all feature vectors collectively constitute a text coding space; calculating a feature distance for each positive and negative text pair in the text coding space; constructing a text intra-model contrast loss based on the feature distance, taking the shortening of the distance of the positive text pairs and the lengthening of the distance of the negative text pairs as the optimization goal, and performing fine-grained alignment training on the target learned model based on the text intra-model contrast loss to obtain a preset graphic-text matching model.

[0092] It should be understood that the positive text pair can refer to a text combination with high semantic similarity, complete consistency of core semantics and extremely small fine-grained difference (without affecting the core scene matching), such as Chinese "a white cat standing on a stone" and "a white kitten standing on a stone", which is used to guide the model to perceive the feature association of semantic consistency. The negative text pair can refer to a text combination with high semantic similarity, similar core semantics but key fine-grained difference (affecting the core scene matching), such as Chinese "a white cat standing on a stone" and "a white cat sitting on a stone" (the difference is the action of "standing" and "sitting", which changes the spatial interaction relationship between the cat and the stone), which is used to guide the model to distinguish the feature boundary of semantic difference.

[0093] In a specific implementation, first, all English and Chinese texts (including short / long texts) are extracted from the target data set, text combinations with consistent core semantics are screened based on a semantic similarity algorithm (such as cosine similarity preliminary screening), for example, around the core scene of "a white cat on a stone", Chinese texts such as "a white cat standing on a stone", "a white kitten standing on a stone" and "a white cat sitting on a stone" and corresponding English texts are screened; then, the text pairs are classified according to whether the fine-grained difference affects the core scene: the positive text pairs (such as "a white cat standing on a stone" and "a white kitten standing on a stone") with unchanged core scene and extremely small difference, and the negative text pairs (such as "a white cat standing on a stone" and "a white cat sitting on a stone") with core scene association but key detail difference, so as to ensure that the positive and negative text pairs focus on fine-grained dimensional difference.

[0094] The filtered positive and negative text pairs (processed separately in Chinese and English) are input into the target post-learning model. The text encoder of the model (which has basic semantic extraction capability after pre-training) performs semantic analysis and encoding on each text. The text is tokenized into initial vectors through the word embedding layer, and the semantic association between words is captured through the Transformer attention mechanism, and finally a fixed-dimensional feature vector (such as 512 dimensions) is output. The feature vectors of all texts together form a text encoding space, for example, the vector V1 corresponds to "a white cat standing on a stone", the vector V2 corresponds to "a white kitten standing on a stone", and the vector V3 corresponds to "a white cat sitting on a stone". V1, V2, and V3 are collectively in the space.

[0095] In the constructed text encoding space, for each set of positive and negative text pairs, the cosine distance is used to calculate the distance between the feature vectors. For the positive text pair (V1 and V2), the cosine distance D_pos (initially 0.4, indicating that the semantics are close but still have optimization space) is calculated. For the negative text pair (V1 and V3), the cosine distance D_neg (initially 0.5, indicating that the semantic difference is not fully distinguished) is calculated. The distance calculation results of all positive and negative text pairs provide quantitative basis for subsequent loss construction.

[0096] Based on the calculated D_pos and D_neg, the TIC Loss function is constructed. The function design follows the principle that "the smaller D_pos and the larger D_neg, the smaller the loss value". The loss function is used as the optimization objective, and the text encoder parameters of the target post-learning model are updated through the backpropagation algorithm to adjust the attention weight, so that the model pays more attention to fine-grained semantic features such as "standing" and "sitting", "cat" and "kitten". After multiple rounds of iterative training (each round traverses all positive and negative text pairs), the TIC Loss value converges (e.g., from the initial 0.6 to 0.2), at which point the model can accurately distinguish fine-grained semantic differences, and finally convert into a preset image-text matching model.

[0097] The embodiment based on fine-grained learning objectives and text-in-model contrastive loss jointly performs fine-grained alignment training on the globally aligned model to obtain a preset image-text matching model, thereby improving the effectiveness of fine-grained alignment training and further improving the model's ability to distinguish semantically similar texts.

[0098] For ease of understanding, the following is illustrated by way of example, but does not limit the present application. As an example, in recent years, CLIP-based visual-language alignment models have achieved strong zero-shot image classification and image-text retrieval capabilities by performing contrastive learning on large-scale image-text pairs. However, existing models perform poorly on fine-grained understanding tasks such as distinguishing similar object attributes, spatial layouts, or semantic nuances. The root cause lies in the fact that the training data is mostly coarse-grained image-short text pairs, and the model tends to learn topic-level alignment, making it difficult to capture the precise correspondence between image regions and text segments. Although there have been attempts to improve, there are still significant limitations: (1) FineCLIP, LongCLIP, and other methods introduce region signals or long texts, but do not systematically address the fine-grained alignment problem in the Chinese-English bilingual scenario; (2) Existing Chinese visual-language models (such as Chinese-CLIP) mainly focus on short text retrieval and lack support for fine-grained semantics such as attributes and spatial relationships; (3) There is a lack of unified training framework and high-quality bilingual dataset, making it difficult for models to achieve robust fine-grained alignment in Chinese-English scenarios.

[0099] Therefore, the related art has obvious deficiencies in supporting Chinese-English bilingual and achieving high-precision fine-grained visual-language alignment. There is an urgent need for a new unified framework that can overcome the dual challenges of language and granularity and achieve precise semantic matching across languages and modalities.

[0100] To solve the above technical problems, the present application proposes FG-CLIP 2 (Fine-Grained CLIP 2), a unified fine-grained visual-language alignment method for Chinese-English bilingual scenarios. This method uses a two-stage progressive training paradigm, combined with large-scale high-quality bilingual data and a new text intra-modal contrastive loss, to achieve precise alignment of images and texts at the region level and semantic level.

[0101] Technical idea: The core of FG-CLIP 2 lies in the two-stage training strategy, which gradually improves the model's ability from global alignment to fine-grained discrimination. The first stage learns the global semantic alignment of images and long and short texts; the second stage introduces fine-grained learning objectives and designs a text intra-modal contrastive loss (Textual Intra-modal Contrastive, TIC Loss) to strengthen the model's ability to distinguish semantically similar texts.

[0102] The specific implementation process is as follows: 1、Bilingual data construction and preprocessing: Construct a large-scale and high-quality Chinese-English image-text pair dataset, containing short and long text descriptions. Clean and align the text to ensure semantic consistency between Chinese and English, supporting fine-grained tasks such as attribute recognition and spatial relationship understanding.

[0103] 2、First stage: Global alignment training. Use image encoder and text encoder to extract image features and text features respectively. For each image, match its corresponding short and long text descriptions simultaneously, and calculate the contrast learning loss. This stage aims to establish coarse-grained and partially fine-grained semantic associations between images and texts in the early stage of training, laying the foundation for subsequent stages.

[0104] 3、Second stage: Fine-grained alignment training. Based on the first stage model, introduce the following fine-grained learning goals: Region-level alignment goal: Use image region candidate boxes and noun phrases in text for local alignment.

[0105] Discriminative learning goal: Introduce attribute-level hard negatives, such as "red apple" and "green apple" as negative pairs, to improve the model's sensitivity to subtle semantic differences.

[0106] Cross-modal ranking goal: Optimize the ranking performance of image-text pairs to improve the fine-grained discrimination ability in retrieval tasks.

[0107] Propose text-in-context contrast loss (TIC Loss): Select text pairs with high semantic similarity from the training set (e.g., "running dog" and "running puppy"). In the text encoding space, close the distance between positive text pairs and push away negative text pairs with high similarity. This loss function directly enhances the text encoder's ability to distinguish subtle semantic differences, thereby improving the accuracy of cross-modal matching.

[0108] 4、Model inference and application: After training, the model can be used for image-text retrieval, zero-shot classification, visual question answering, and other tasks. Given a query text, the model calculates its similarity with candidate images and returns the most matching result. Due to fine-grained training, the model can accurately distinguish subtle differences such as "person with hat" and "person without hat", and performs well in both Chinese and English scenarios.

[0109] The embodiment illustrates that, in the Chinese image-text retrieval task, the FG-CLIP 2 model can not only match the image containing the "cat" and "stone", but also accurately exclude the interference items such as "black cat" or "cat on the grass" through the fine-grained training and TIC loss in the second stage, to realize high-precision retrieval. Similarly, in the English task, the model can also effectively distinguish "a man in a red shirt" from "a man in a blue shirt".

[0110] Compared with the related art, the FG-CLIP 2 method provided in the present application has the following remarkable beneficial effects: 1. Realize bilingual fine-grained alignment: for the first time, a unified framework is constructed to support fine-grained visual-linguistic understanding in Chinese and English scenarios, filling the gap in Chinese fine-grained multi-modal research; 2. Two-stage training improves performance: through the progressive strategy of "global alignment → fine-grained fine-tuning", the model learns more stably, and the fine-grained discrimination ability is significantly enhanced; 3. TIC loss improves text discrimination: the proposed text-in-context contrastive loss (TIC Loss) effectively enhances the model's ability to distinguish similar texts.

[0111] It should be noted that the data collection, label management, rule setting or push decision involved in the present application are designed to solve technical problems in cooperation with other technical features, and do not involve or support any illegal behavior. Any data processing behavior that may violate laws and regulations (such as unauthorized collection of private data, generation of discriminatory labels, setting of unfair rules, and push of illegal information) does not fall within the protection scope of the technical solution of the present application. Of course, the user data in the present application will be processed by encryption, anonymization or de-identification before storage, to ensure the security of the user data.

[0112] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image-text matching method of the present application. Further simple transformations based on this technical concept are within the protection scope of the present application.

[0113] The present application also provides an image-text matching device, which is described in detail in the Figure 4 The image-text matching device comprises: A text input module 10 is configured to input the query text into a preset image-text matching model in response to the input query text. The feature extraction module 20 is configured to extract image features of the candidate images and text features of the query text respectively by using the preset image-text matching model, wherein the preset image-text matching model is a model obtained by fine-grained alignment training based on a text-in-model contrast loss, and the fine-grained alignment training based on the text-in-model contrast loss is used to enhance the ability of the model to distinguish between semantically similar texts. The image-text matching module 30 is configured to determine a target image matching the query text based on the image features and the text features.

[0114] The image-text matching device provided in the present application adopts the image-text matching method in the above embodiments, and can solve the technical problem that related image-text matching models perform poorly in processing fine-grained understanding tasks, thereby having low fine-grained matching precision. Compared with the prior art, the image-text matching device provided in the present application has the same beneficial effects as the image-text matching method provided in the above embodiments, and other technical features in the image-text matching device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0115] The present application provides an image-text matching device, which comprises at least one processor and a memory connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image-text matching method in the above embodiment one.

[0116] Reference will be made to the following description of the drawings Figure 5 which shows a structural schematic diagram of an image-text matching device suitable for implementing the embodiments of the present application. The image-text matching device in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 5 The image-text matching device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0117] As Figure 5As shown, the text-image matching device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a ROM (Read Only Memory) 1002 or programs loaded from a storage device 1003 into a RAM (Random Access Memory) 1004. Various programs and data required for the operation of the text-image matching device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. In general, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the text-image matching device to communicate with other devices wirelessly or by wire to exchange data. Although the text-image matching device with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0118] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0119] The text-image matching device provided in the present application adopts the text-image matching method in the above-mentioned embodiments, which can solve the technical problem that related text-image matching models perform poorly in processing fine-grained understanding tasks, thereby having low fine-grained matching precision. Compared with the prior art, the text-image matching device provided in the present application has the same beneficial effects as the text-image matching method provided in the above-mentioned embodiments, and other technical features in the text-image matching device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0120] It should be understood that various parts of the present application can be realized in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0121] The above merely provides a part of the embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0122] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer programs) for performing the image-text matching method in the above embodiments.

[0123] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for performing the above image-text matching method, and can solve the technical problem that related image-text matching models perform poorly in processing fine-grained understanding tasks, thereby having low fine-grained matching precision. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the image-text matching method provided by the above embodiments, which will not be described here.

[0124] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the image-text matching method as described above.

[0125] The computer program product provided by the present application can solve the technical problem that related image-text matching models perform poorly in processing fine-grained understanding tasks, thereby having low fine-grained matching precision. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the image-text matching method provided by the above embodiments, which will not be described here.

[0126] The above only describes some embodiments of the present application, and does not limit the scope of the present application, and any equivalent structural transformation made by using the contents of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the protection scope of the present application.

[0127] The present application discloses A1, an image-text matching method, which comprises: in response to an input query text, inputting the query text into a preset image-text matching model; extracting image features of the candidate image and text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained through fine-grained alignment training based on a text-in-model contrast loss, and the fine-grained alignment training based on the text-in-model contrast loss is used to enhance the ability of the model to distinguish semantically similar texts; determining a target image matching the query text based on the image features and the text features.

[0128] A2. The image-text matching method of A1, before inputting the query text into the preset image-text matching model in response to the input query text, further comprising: constructing a target data set, wherein the target data set comprises short text descriptions, long text descriptions, and corresponding sample images; performing global alignment training on an initial image-text matching model based on the target data set to obtain a globally aligned model; performing fine-grained alignment training on the globally aligned model based on the target data set to obtain the preset image-text matching model, wherein the fine-grained alignment training comprises fine-grained alignment training based on a text-in-model contrast loss.

[0129] A3. The image-text matching method of A2, wherein performing fine-grained alignment training on the globally aligned model based on the target data set to obtain the preset image-text matching model comprises: performing fine-grained alignment training on the globally aligned model based on the target data set and a fine-grained learning target to obtain a target learned model, wherein the fine-grained learning target comprises at least one of a region-level alignment target, a discriminative learning target, and a cross-modal ranking target; performing fine-grained alignment training on the target learned model based on the target data set and a text-in-model contrast loss to obtain the preset image-text matching model.

[0130] A4. The image-text matching method of A3, wherein performing fine-grained alignment training on the target learned model based on the target data set and a text-in-model contrast loss to obtain the preset image-text matching model comprises: selecting text pairs with high semantic similarity from the target data set and dividing the text pairs into positive text pairs and negative text pairs; inputting the positive text pairs and the negative text pairs into the target learned model, and converting each text into a fixed-dimensional feature vector through the target learned model, all feature vectors together forming a text encoding space; calculating feature distances for each group of positive and negative text pairs in the text encoding space; Construct a text-in-model contrast loss based on the feature distance, so as to narrow the distance of positive text pairs and push away the distance of negative text pairs as an optimization goal, and perform fine-grained alignment training on the target learned model based on the text-in-model contrast loss, to obtain a preset image-text matching model.

[0131] A5, the image-text matching method of A3, the fine-grained learning goal includes a region-level alignment goal, and the fine-grained alignment training of the global alignment model based on the target data set and the fine-grained learning goal obtains a target learned model, including: extract image region candidate boxes from the sample image, and extract text noun phrases from the short text description and the long text description; extract fine-grained visual features from the image region candidate boxes and fine-grained semantic features from the text noun phrases through the global alignment model; perform region-level alignment training on the global alignment model based on the fine-grained visual features and the fine-grained semantic features, to obtain a target learned model.

[0132] A6, the image-text matching method of A3, the fine-grained learning goal includes a discriminative learning goal, and the fine-grained alignment training of the global alignment model based on the target data set and the fine-grained learning goal obtains a target learned model, including: screen attribute-level hard-negative sample pairs from the target data set according to the screening principle of consistent theme and different attributes; input the attribute-level hard-negative sample pairs into the global alignment model, and calculate a discriminative learning loss of the attribute-level hard-negative sample pairs according to the output result of the global alignment model; perform discriminative learning training on the global alignment model based on the discriminative learning loss, to obtain a target learned model.

[0133] A7, the image-text matching method of A3, the fine-grained learning goal includes a cross-modal sorting goal, and the fine-grained alignment training of the global alignment model based on the target data set and the fine-grained learning goal obtains a target learned model, including: construct at least one group of training samples with the target data set as a data source, wherein each group of training samples includes a core text, an image matching the core text, an image partially matching the core text, and an image not matching the core text; calculate the similarity scores of each group of image-text pairs in the training samples, and sort each group of image-text pairs according to the similarity scores to obtain a similarity sorting result; Construct a ranking loss according to the similarity ranking result, and perform cross-modal ranking training on the global alignment model based on the ranking loss to obtain a target learned model.

[0134] A8. The image-text matching method of A2, wherein the global alignment training of the initial image-text matching model based on the target data set to obtain a global alignment model comprises: extracting global visual features of the sample image by the initial image-text matching model, and extracting global semantic features of the short text description and the long text description; calculating a contrast learning loss based on the global visual features and the global semantic features; performing global alignment training of the initial image-text matching model according to the contrast learning loss to obtain a global alignment model.

[0135] A9. The image-text matching method of A2, wherein the target data set comprises: constructing an English-Chinese image-text pair data set, wherein the English-Chinese image-text pair data set comprises English-Chinese short text descriptions, English-Chinese long text descriptions, and corresponding sample images; performing data cleaning on the texts in the English-Chinese image-text pair data set to obtain a cleaned data set; performing semantic alignment on the Chinese texts and the English texts in the cleaned data set to obtain a target data set.

[0136] A10. The image-text matching method of A9, wherein the semantic alignment of the Chinese texts and the English texts in the cleaned data set to obtain a target data set comprises: analyzing semantic differences between the Chinese texts and the English texts in the cleaned data set by a preset large language model; performing semantic alignment on the Chinese texts and the English texts by the preset large language model based on the semantic differences to obtain a target data set.

[0137] A11. The image-text matching method of any one of A1 to A10, wherein the determination of a target image matching the query text based on the image features and the text features comprises: calculating similarities between the query text and each candidate image according to the image features and the text features; ranking each candidate image based on the similarities, and determining a target image matching the query text according to the ranking result.

[0138] The application further discloses B12, an image-text matching device, comprising: a text input module configured to input a query text into a preset image-text matching model in response to the input query text; a feature extraction module configured to extract image features of the candidate images and text features of the query text respectively by using the preset image-text matching model, wherein the preset image-text matching model is a model obtained by fine-grained alignment training based on a text-in-model contrast loss, and the fine-grained alignment training based on the text-in-model contrast loss is used to enhance the ability of the model to distinguish between semantically similar texts; an image-text matching module configured to determine target images matching the query text based on the image features and the text features.

[0139] B13. The image-text matching apparatus of B12, further comprising: a model training module configured to construct a target dataset, wherein the target dataset comprises short text descriptions, long text descriptions, and corresponding sample images; perform global alignment training on an initial image-text matching model based on the target dataset to obtain a globally aligned model; and perform fine-grained alignment training on the globally aligned model based on the target dataset to obtain a preset image-text matching model, wherein the fine-grained alignment training comprises fine-grained alignment training based on a text-in-model contrast loss.

[0140] B14. The image-text matching apparatus of B13, wherein the model training module is further configured to perform fine-grained alignment training on the globally aligned model based on the target dataset and a fine-grained learning target to obtain a target learned model, wherein the fine-grained learning target comprises at least one of a region-level alignment target, a discriminative learning target, and a cross-modal ranking target; and perform fine-grained alignment training on the target learned model based on the target dataset and a text-in-model contrast loss to obtain the preset image-text matching model.

[0141] B15. The image-text matching apparatus of B14, wherein the model training module is further configured to select text pairs with high semantic similarity from the target dataset and divide the text pairs into positive text pairs and negative text pairs; input the positive text pairs and the negative text pairs into the target learned model, and convert each text into a feature vector of a fixed dimension by using the target learned model, wherein all the feature vectors collectively form a text encoding space; calculate a feature distance for each group of positive and negative text pairs in the text encoding space; construct a text-in-model contrast loss based on the feature distances, wherein the optimization objective is to reduce the distance between positive text pairs and increase the distance between negative text pairs; and perform fine-grained alignment training on the target learned model based on the text-in-model contrast loss to obtain the preset image-text matching model.

[0142] B16. The image-text matching apparatus of B14, the fine-grained learning target comprises a region-level alignment target, and the model training module is further configured to extract image region candidate boxes from the sample image, and extract text noun phrases from the short text description and the long text description; extract fine-grained visual features from the image region candidate boxes and extract fine-grained semantic features from the text noun phrases through the global alignment post-model; and perform region-level alignment training on the global alignment post-model based on the fine-grained visual features and the fine-grained semantic features to obtain a target learned model.

[0143] B17. The image-text matching apparatus of B14, the fine-grained learning target comprises a discriminative learning target, and the model training module is further configured to screen attribute-level hard-negative sample pairs from the target data set according to a screening principle of consistent theme and different attributes; input the attribute-level hard-negative sample pairs into the global alignment post-model, and calculate a discriminative learning loss of the attribute-level hard-negative sample pairs according to an output result of the global alignment post-model; and perform discriminative learning training on the global alignment post-model based on the discriminative learning loss to obtain a target learned model.

[0144] The present application also discloses C18, an image-text matching device, comprising a memory, a processor, and an image-text matching program stored in the memory and executable on the processor, and the image-text matching program, when executed by the processor, implements the image-text matching method as described above.

[0145] The present application also discloses D19, a storage medium, having an image-text matching program stored thereon, and the image-text matching program, when executed by a processor, implements the image-text matching method as described above.

[0146] The present application also discloses E20, a computer program product, comprising an image-text matching program, and the image-text matching program, when executed by a processor, implements the image-text matching method as described above.

Claims

1. A method of image-text matching, characterized by, The image-text matching method comprises: in response to an input query text, inputting the query text into a preset image-text matching model; extracting image features of a candidate image and text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained through fine-grained alignment training based on a text-in-text contrast loss, and the fine-grained alignment training based on the text-in-text contrast loss is used to enhance the ability of the model to distinguish between semantically similar texts; determining a target image matching the query text based on the image features and the text features.

2. The graph matching method of claim 1, wherein, Before the step of inputting the query text into the preset image-text matching model in response to the input query text, the method further comprises: constructing a target data set, wherein the target data set comprises short text descriptions, long text descriptions, and corresponding sample images; performing global alignment training on an initial image-text matching model based on the target data set to obtain a globally aligned model; performing fine-grained alignment training on the globally aligned model based on the target data set to obtain the preset image-text matching model, wherein the fine-grained alignment training comprises fine-grained alignment training based on a text-in-text contrast loss.

3. The graph matching method of claim 2, wherein, The fine-grained alignment training based on the target data set and the fine-grained learning target on the globally aligned model to obtain the target learned model comprises: performing fine-grained alignment training on the globally aligned model based on the target data set and a fine-grained learning target to obtain a target learned model, wherein the fine-grained learning target comprises at least one of a region-level alignment target, a discriminative learning target, and a cross-modal ranking target; performing fine-grained alignment training on the target learned model based on the target data set and a text-in-text contrast loss to obtain the preset image-text matching model.

4. The graph matching method of claim 3, wherein, The fine-grained alignment training based on the target data set and the text-in-text contrast loss on the target learned model to obtain the preset image-text matching model comprises: selecting text pairs with high semantic similarity from the target data set and dividing the text pairs into positive text pairs and negative text pairs; inputting the positive text pairs and the negative text pairs into the target learned model, and converting each text into a fixed-dimensional feature vector through the target learned model, all feature vectors together forming a text encoding space; calculating feature distances for each group of positive and negative text pairs in the text encoding space; constructing a text-in-text contrast loss based on the feature distances, taking the goal of shortening the distance between positive text pairs and lengthening the distance between negative text pairs as an optimization goal, and performing fine-grained alignment training on the target learned model based on the text-in-text contrast loss to obtain the preset image-text matching model.

5. The graph matching method of claim 3, wherein, The fine-grained learning target comprises a region-level alignment target, and the fine-grained alignment training based on the target data set and the fine-grained learning target on the globally aligned model to obtain the target learned model comprises: extracting image region candidate boxes from the sample images and text noun phrases from the short text descriptions and the long text descriptions; extracting fine-grained visual features from the image region candidate box and extracting fine-grained semantic features from the text noun phrase through the global alignment model; performing region-level alignment training on the global alignment model based on the fine-grained visual features and the fine-grained semantic features to obtain a target learned model.

6. The graph matching method of claim 3, wherein, The fine-grained learning target includes a discriminative learning target, and performing fine-grained alignment training on the global alignment model based on the target data set and the fine-grained learning target to obtain a target learned model includes: filtering attribute-level hard-negative sample pairs from the target data set according to a filtering principle of consistent theme and different attributes; inputting the attribute-level hard-negative sample pairs into the global alignment model and calculating a discriminative learning loss of the attribute-level hard-negative sample pairs according to an output result of the global alignment model; performing discriminative learning training on the global alignment model based on the discriminative learning loss to obtain a target learned model.

7. A picture-text matching apparatus characterized by comprising: The image-text matching apparatus includes: a text input module configured to input a query text into a preset image-text matching model in response to the input query text; a feature extraction module configured to extract image features of a candidate image and text features of the query text through the preset image-text matching model, wherein the preset image-text matching model is a model obtained through fine-grained alignment training based on a text-in-model contrast loss, and the fine-grained alignment training based on the text-in-model contrast loss is used to enhance the ability of the model to distinguish between semantically similar texts; an image-text matching module configured to determine a target image matching the query text based on the image features and the text features.

8. An image-text matching apparatus characterized by comprising: The image-text matching apparatus includes a memory, a processor, and an image-text matching program stored on the memory and executable on the processor, and the image-text matching program, when executed by the processor, implements the image-text matching method of any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium has an image-text matching program stored thereon, and the image-text matching program, when executed by the processor, implements the image-text matching method of any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program product includes an image-text matching program, and the image-text matching program, when executed by the processor, implements the image-text matching method of any one of claims 1 to 6.

Images