Image-text retrieval method based on multi-granularity alignment

Abstract

The invention discloses an image-text retrieval method based on multi-granularity alignment, and the method comprises the steps: carrying out the coding processing of image data and text data through a feature extraction module, and extracting an image global feature vector, an image patch feature set, a sentence-level feature vector and a word-level feature set; obtaining coarse-grained similarity data of a matching relationship between the image overall semantics and the text overall semantics; constructing a patch-word similarity matrix; multi-granularity image-text similarity fusing global and local information is obtained, and a matching result with the highest similarity serves as an image-text retrieval result; obtaining a trained image-text retrieval model based on the multi-granularity image-text similarity; and obtaining a retrieval result by utilizing the trained image-text retrieval model based on the multi-granularity image-text similarity. According to the method, the accuracy and generalization ability of image-text retrieval are remarkably improved, and the method has a positive promotion effect in various fields such as automatic question-answering systems, intelligent transportation and medical image analysis.

Description

Technical Field

[0001] This invention relates to a text and image retrieval method based on multi-granularity alignment, belonging to the field of text and image retrieval. Background Technology

[0002] Image-text retrieval is a crucial task in computer vision and natural language processing, aiming to enable mutual retrieval of images and text based on semantic relationships. Traditional image-text retrieval methods typically match global features of images and text, such as calculating similarity after extracting global feature vectors from both. While effective in certain scenarios, this approach tends to overlook local details in images and key phrases in text, leading to suboptimal retrieval performance in complex environments. Furthermore, most existing methods rely on a single granularity alignment strategy, failing to fully utilize matching information between images and text at different granularities. Therefore, image-text retrieval methods based on multi-granularity alignment have become a research hotspot.

[0003] However, existing multi-granularity alignment methods still face some challenges, such as how to effectively capture local features while preserving global information and how to solve the fusion problem between information of different granularities. Therefore, proposing a novel image and text retrieval method based on multi-granularity alignment to further improve the performance of image and text retrieval is an important direction in current research. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-granularity alignment-based image and text retrieval method to solve the problem in the prior art of how to effectively integrate information of different granularities of images and text to improve retrieval performance.

[0005] The technical solution of this invention is:

[0006] A text and image retrieval method based on multi-granularity alignment includes the following steps:

[0007] Step 1: The feature extraction module encodes the image data and text data respectively, and extracts the global feature vector of the image, the image patch feature set, the sentence-level feature vector, and the word-level feature set;

[0008] Step 2: Based on the global feature vector of the image and the sentence-level feature vector obtained in Step 1, the coarse-grained calculation module performs similarity calculation to obtain coarse-grained similarity data on the overall semantic matching relationship between the image and the overall semantic matching relationship between the text. This data is used to constrain and guide subsequent fine-grained similarity calculations.

[0009] Step 3, based on the image patch feature set and the word-level feature set obtained in step 1, the similarity between each image patch and text word is calculated by the similarity calculation module, a patch-word similarity matrix is constructed, and the fine-grained semantic association relationship between the image and the text is described and determined based on the similarity matrix;

[0010] Step 4, based on the patch-word similarity matrix obtained in step 3 and combined with the coarse-grained similarity data in step 2, the local similarity between the image and the text is processed by the feature fusion module. Bidirectional interactive aggregation and normalization, get multi-granularity image-text similarity fused with global and local information, and the matching result with the highest similarity is taken as the image-text retrieval result;

[0011] Step 5, the feature extraction module, the coarse-grained calculation module, the similarity calculation module and the feature fusion module constitute the image-text retrieval model based on multi-granularity image-text similarity, an alignment loss function for constraining image-text matching relationship is constructed, and the image-text retrieval model based on multi-granularity image-text similarity is optimized through the alignment loss function. Get the trained image-text retrieval model based on multi-granularity image-text similarity;

[0012] Step 6, in the image-text retrieval stage, for the input query image or query text, the trained image-text retrieval model based on multi-granularity image-text similarity obtained in step 5 is used to obtain the retrieval result.

[0013] Further, step 1 is specifically,

[0014] Step 1.1, obtain image data and text data corresponding to the image data for image-text cross-modal retrieval;

[0015] Step 1.2, encode the image data, divide the input image into multiple non-overlapping image patches, and input the image patches into the image encoder for feature extraction to obtain the image global feature vector and the image patch feature set , wherein, represents a C-dimensional real vector space, represents the first to the Mth image patch feature vector, represents that the set contains M patches, and each patch is represented by a C-dimensional vector; Step 1.3, encode the text data, convert the text data into a text sequence composed of words, and input the text sequence into the text encoder for feature extraction to obtain the sentence-level feature vector and the word-level feature set , wherein s represents the sentence-level feature vector. represents a C-dimensional real vector space; represents the first to the Nth word-level feature vector, represents a set containing N words, each word represented by a C-dimensional vector.

[0016] Further, step 2, specifically,

[0017] Step 2.1, for image samples and sentence samples , cosine similarity calculation is performed based on the image global feature vectors and sentence-level feature vectors obtained in step 1:

[0018] ,

[0019] wherein sim represents a similarity function for measuring the degree of semantic matching between images and text; cos represents a cosine similarity function for calculating the directional similarity of two feature vectors in the feature space; g represents a global granularity for distinguishing similarity calculations at different granularity levels; represents the image global feature vector extracted by the image encoder for the i-th image sample ; represents the image global feature vector extracted by the image encoder for the i-th image sample ; represents the sentence-level feature vector extracted by the text encoder for the i-th sentence sample ;

[0020] Step 2.2, organize the multiple image-text coarse-grained similarity obtained in step 2.1 to obtain coarse-grained similarity data of the overall semantic matching relationship between images and text.

[0021] Further, step 3, specifically,

[0022] Step 3.1, similarity calculation is performed on the image patch feature set and word-level feature set obtained in step 1 to obtain a patch-word similarity matrix. Wherein, represents the i-th image patch ; represents the i-th image patch ; represents the i-th image patch ;

[0023] Step 3.2, based on the patch-word similarity matrix, analyze the similarity distribution between each image patch and text word, determine the image patch features with a text word-level semantic association degree higher than a set value, and take the image patch features as local matching features.

[0024] Further, step 4, specifically,

[0025] Step 4.1: Based on the patch-word similarity matrix obtained in Step 3, and combined with the local matching features obtained in Step 3.2, the similarity values ​​corresponding to the local matching features in the patch-word similarity matrix are filtered and retained to obtain the fine-grained similarity relationship between each image patch and each text word, which serves as the input data for subsequent bidirectional interactive aggregation processing.

[0026] Step 4.2: In the matching direction of the image relative to the text, for each word in the text word-level feature set, based on the fine-grained similarity relationship obtained in Step 4.1, calculate the similarity distribution between the word and each image patch in the image, and obtain the first weight distribution of the word on different image patches through normalization; use the first weight distribution to perform weighted aggregation of the corresponding patch-word similarity to obtain the first matching score of each word on the whole image; perform weighted summation of the first matching scores corresponding to all words to obtain the fine-grained similarity result of the image relative to the text in the first direction.

[0027] Step 4.3: In the matching direction of text relative to image, for each image patch in the image patch feature set, based on the fine-grained similarity relationship obtained in Step 4.1, calculate the similarity distribution between the image patch and each word in the text, and obtain the second weight distribution of the image patch on different words through normalization; use the second weight distribution to perform weighted aggregation of the corresponding patch-word similarity to obtain the second matching score of each image patch on the entire text; perform weighted summation of the second matching scores corresponding to all image patches to obtain the second-direction fine-grained similarity result of text relative to image;

[0028] Step 4.4: The fine-grained similarity result of the image relative to the text in the first direction obtained in Step 4.2 is fused with the fine-grained similarity result of the text relative to the image in the second direction obtained in Step 4.3, as follows:

[0029] ,

[0030] in, This represents the fine-grained fusion result obtained by comprehensively considering the bidirectional matching relationship between the i-th image sample and the j-th text sample at a fine-grained level; This represents the fine-grained similarity result between the i-th image sample and the j-th text sample in the matching direction of the image relative to the text; This represents the fine-grained similarity result between the i-th image sample and the j-th text sample in the matching direction of the text relative to the image;

[0031] Step 4.5: Combining the coarse-grained similarity data obtained in Step 2, the fine-grained fusion result obtained in Step 4.4 is constrained using a global semantic constraint method to obtain a multi-granularity image-text similarity that integrates global and local information, and the matching result with the highest similarity is taken as the image-text retrieval result.

[0032] Furthermore, in step 5, an alignment loss function is constructed to constrain the image-text matching relationship, specifically as follows:

[0033] Based on multi-granularity image-text similarity data, an alignment loss function is constructed. This function considers the similarity constraints in both the image-to-text direction and the text-to-image direction, aiming to narrow the similarity of matching image-text pairs and widen the similarity of mismatched pairs. The alignment loss function L is:

[0034] ,

[0035] in, and Let represent the contrast loss from image to text and from text to image, respectively, and define them using the following formulas:

[0036] ,

[0037] ,

[0038] in, Indicates the similarity of matched pairs, off-diagonal. and Indicates a mismatched image / text pair; This represents the total number of samples within a batch. Indicates the index of the sample currently being processed. Indicates the index of the comparison sample for the summation calculation.

[0039] The beneficial effects of this invention are as follows: Compared with existing technologies, this multi-granularity alignment-based image-text retrieval method, by introducing a multi-level alignment mechanism of coarse-grained (global feature vector matching) and fine-grained (patch and word matching), can more comprehensively model the semantic relationship between images and text. Simultaneously, through interactive similarity aggregation and normalization processing, it effectively eliminates scale differences between different samples, improving the stability of matching results. Furthermore, by combining a specially designed multi-granularity contrastive loss function, an image-text retrieval model based on multi-granularity image-text similarity is obtained, thereby significantly improving the accuracy and generalization ability of image-text retrieval, and playing a positive role in various fields such as automatic question answering systems, intelligent transportation, and medical image analysis. Attached Figure Description

[0040] Figure 1This is a flowchart illustrating the image and text retrieval method based on multi-granularity alignment according to an embodiment of the present invention. Detailed Implementation

[0041] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0042] This embodiment provides a text and image retrieval method based on multi-granularity alignment, including the following steps:

[0043] Step 1: The feature extraction module encodes the image data and text data respectively, and extracts the global feature vector of the image, the image patch feature set, the sentence-level feature vector, and the word-level feature set.

[0044] Step 1.1: Obtain image data and corresponding text data for cross-modal image-text retrieval;

[0045] Step 1.2: Encode the image data by dividing the input image into multiple non-overlapping image patches. Input these patches into an image encoder for feature extraction to obtain the global feature vector of the image. and image patch feature set ,in, Represents a C-dimensional real vector space. This represents the feature vector of the first to the Mth graphic patches. The set contains M patches, each patch being represented by a C-dimensional vector. Step 1.3: Encode the text data, converting it into a text sequence composed of words, and input the text sequence into a text encoder for feature extraction to obtain sentence-level feature vectors. and word-level feature set Where s represents the sentence-level feature vector; Represents a C-dimensional real vector space; This represents the word-level feature vector from the 1st to the Nth word. The set contains N words, each represented by a C-dimensional vector.

[0046] Step 2: Based on the global image feature vector and sentence-level feature vector obtained in Step 1, the coarse-grained calculation module performs similarity calculation to obtain coarse-grained similarity data on the overall semantic matching relationship between the image and the text, which is used to constrain and guide subsequent fine-grained similarity calculation.

[0047] Step 2.1, for image samples Sentence Sample Cosine similarity is calculated on the global feature vector and sentence-level feature vector of the image obtained in step 1:

[0048] ,

[0049] Where sim represents the similarity function, which measures the semantic matching degree between images and text; cos represents the cosine similarity function, which calculates the directional similarity between two feature vectors in the feature space; g represents the global granularity, which distinguishes the similarity calculation at different granularity levels. This indicates that the image encoder processed the first... Image samples The extracted global feature vector of the image; This indicates that the text encoder is used for the first... Sentence samples Extracted sentence-level feature vectors;

[0050] Step 2.2: Organize the multiple image-text coarse-grained similarities obtained in Step 2.1 to obtain coarse-grained similarity data of the overall semantic matching relationship between the image and the overall semantic matching relationship between the text.

[0051] Step 3: Based on the image patch feature set and word-level feature set obtained in Step 1, the similarity calculation module performs similarity calculation on each image patch and text word, constructs a patch-word similarity matrix, and characterizes and determines the fine-grained semantic relationship between the image and the text based on the similarity matrix.

[0052] Step 3.1: Calculate the similarity between the image patch feature set and the text word-level feature set obtained in Step 1 to obtain... The patch-word similarity matrix. express An image patch, express Each text word;

[0053] Step 3.2: Based on the patch-word similarity matrix, analyze the similarity distribution between each image patch and text word, determine the image patch features with a semantic correlation higher than the set value with the text word level, and use the image patch features as local matching features.

[0054] Step 4: Based on the patch-word similarity matrix obtained in Step 3 and combined with the coarse-grained similarity data in Step 2, the feature fusion module performs bidirectional interactive aggregation and normalization on the local similarity between images and text to obtain a multi-granular image-text similarity that integrates global and local information. The matching result with the highest similarity is then output as the image-text retrieval result.

[0055] Step 4.1: Based on the patch-word similarity matrix obtained in Step 3, and combined with the local matching features obtained in Step 3.2, the similarity values ​​corresponding to the local matching features in the patch-word similarity matrix are filtered and retained to obtain the fine-grained similarity relationship between each image patch and each text word, which serves as the input data for subsequent bidirectional interactive aggregation processing.

[0056] Step 4.2: In the matching direction of the image relative to the text, for each word in the text word-level feature set, based on the fine-grained similarity relationship obtained in Step 4.1, calculate the similarity distribution between the word and each image patch in the image, and obtain the first weight distribution of the word on different image patches through normalization; use the first weight distribution to perform weighted aggregation of the corresponding patch-word similarity to obtain the first matching score of each word on the whole image; perform weighted summation of the first matching scores corresponding to all words to obtain the fine-grained similarity result of the image relative to the text in the first direction.

[0057] Step 4.3: In the matching direction of text relative to image, for each image patch in the image patch feature set, based on the fine-grained similarity relationship obtained in Step 4.1, calculate the similarity distribution between the image patch and each word in the text, and obtain the second weight distribution of the image patch on different words through normalization; use the second weight distribution to perform weighted aggregation of the corresponding patch-word similarity to obtain the second matching score of each image patch on the entire text; perform weighted summation of the second matching scores corresponding to all image patches to obtain the second-direction fine-grained similarity result of text relative to image;

[0058] Step 4.4: The fine-grained similarity result of the image relative to the text in the first direction obtained in Step 4.2 is fused with the fine-grained similarity result of the text relative to the image in the second direction obtained in Step 4.3, as follows:

[0059] ,

[0060] in, This represents the fine-grained fusion result obtained by comprehensively considering the bidirectional matching relationship between the i-th image sample and the j-th text sample at a fine-grained level; This represents the fine-grained similarity result between the i-th image sample and the j-th text sample in the matching direction of the image relative to the text; This represents the fine-grained similarity result between the i-th image sample and the j-th text sample in the matching direction of the text relative to the image;

[0061] Step 4.5: Combining the coarse-grained similarity data obtained in Step 2, the fine-grained fusion result obtained in Step 4.4 is constrained using the existing global semantic constraint method to obtain a multi-granularity image-text similarity that integrates global and local information, and the matching result with the highest similarity is taken as the image-text retrieval result.

[0062] Step 5: The feature extraction module, coarse-grained calculation module, similarity calculation module, and feature fusion module constitute a text-image retrieval model based on multi-granularity text-image similarity. An alignment loss function is constructed to constrain the text-image matching relationship. The text-image retrieval model based on multi-granularity text-image similarity is optimized through the alignment loss function to obtain the trained text-image retrieval model based on multi-granularity text-image similarity.

[0063] Step 5.1: Based on the multi-granularity image-text similarity data, construct an alignment loss function. This alignment loss function considers both the image-to-text direction and the text-to-image direction similarity constraints, used to bring matching image-text pairs closer together and widen the similarity of mismatched image-text pairs. The loss function L is:

[0064] ,

[0065] in, and Let represent the contrast loss from image to text and from text to image, respectively, and define them using the following formulas:

[0066] ,

[0067] ,

[0068] in, Indicates the similarity of matched pairs, off-diagonal. and Indicates a mismatched image / text pair; This represents the total number of samples within a batch. Indicates the index of the sample currently being processed. Indicates the index of the comparison sample for the summation calculation.

[0069] Step 6: In the image and text retrieval stage, for the input query image or text, the retrieval results are obtained using the trained image and text retrieval model based on multi-granularity image and text similarity obtained in Step 5.

[0070] This multi-granularity alignment-based image-text retrieval method, compared to existing technologies, introduces a multi-level alignment mechanism of coarse-grained (global feature vector matching) and fine-grained (patch and word matching), enabling a more comprehensive modeling of the semantic relationships between images and text. Simultaneously, interactive similarity aggregation and normalization effectively eliminate scale differences between different samples, improving the stability of matching results. Furthermore, by combining a specially designed multi-granularity contrastive loss function, an image-text retrieval model based on multi-granularity image-text similarity is obtained, significantly improving the accuracy and generalization ability of image-text retrieval, and playing a positive role in various fields such as automatic question answering systems, intelligent transportation, and medical image analysis.

[0071] This multi-granularity alignment-based image-text retrieval method first encodes the input image and text data, extracting global image feature vectors, image patch feature sets, and sentence-level and word-level feature vectors from the text to construct a multi-layered image-text feature representation. Then, similarity is calculated based on the global and sentence-level feature vectors to obtain coarse-grained image-text similarity data representing the overall semantic matching relationship between the image and text. Further, similarity is calculated for each image patch feature and each text word-level feature to construct a patch-word similarity matrix, used to characterize the fine-grained semantic association between the image and text. Based on the patch-word similarity matrix, bidirectional interactive aggregation and normalization are performed, and the fine-grained matching results are constrained by the coarse-grained similarity data to obtain a multi-granularity image-text similarity that integrates global and local information. Finally, an alignment loss function is constructed based on the multi-granularity image-text similarity data, and the model parameters are adjusted by optimizing the loss function to obtain an image-text retrieval model based on multi-granularity image-text similarity for image-text retrieval. During the image and text retrieval stage, this model is used to calculate the multi-granularity image and text similarity between the query image or query text and the matching text or matching image, and outputs the matching result with the highest similarity.

[0072] This invention achieves semantic alignment between images and text in complex scenarios by jointly modeling coarse-grained global information and fine-grained local information, thereby improving retrieval accuracy. Especially in scenarios with short texts or rich local semantic information, it can more comprehensively capture the semantic correspondence between images and text. This invention does not rely on large-scale external knowledge bases and maintains high accuracy and robustness in multiple fields of application. It is applicable to fields such as image and text retrieval, automatic question answering, intelligent transportation, and medical image analysis, and has broad application value.

[0073] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A text and image retrieval method based on multi-granularity alignment, characterized in that: Includes the following steps, Step 1: The feature extraction module encodes the image data and text data respectively, and extracts the global feature vector of the image, the image patch feature set, the sentence-level feature vector, and the word-level feature set; Step 2: Based on the global image feature vector and sentence-level feature vector obtained in Step 1, the coarse-grained calculation module performs similarity calculation to obtain coarse-grained similarity data on the overall semantic matching relationship between the image and the text, which is used to constrain and guide subsequent fine-grained similarity calculations. Step 3: Based on the image patch feature set and word-level feature set obtained in Step 1, the similarity calculation module calculates the similarity between each image patch and text word, constructs a patch-word similarity matrix, and characterizes and determines the fine-grained semantic relationship between the image and the text based on the similarity matrix. Step 4: Based on the patch-word similarity matrix obtained in Step 3 and combined with the coarse-grained similarity data in Step 2, the feature fusion module performs bidirectional interactive aggregation and normalization on the local similarity between the image and the text to obtain a multi-granular image-text similarity that integrates global and local information, and takes the matching result with the highest similarity as the image-text retrieval result. Step 5: The feature extraction module, coarse-grained calculation module, similarity calculation module, and feature fusion module constitute a text-image retrieval model based on multi-granularity text-image similarity. An alignment loss function is constructed to constrain the text-image matching relationship. The text-image retrieval model based on multi-granularity text-image similarity is optimized through the alignment loss function to obtain the trained text-image retrieval model based on multi-granularity text-image similarity. Step 6: In the image and text retrieval stage, for the input query image or query text, the retrieval results are obtained using the trained image and text retrieval model based on multi-granularity image and text similarity obtained in Step 5.

2. The image and text retrieval method based on multi-granularity alignment as described in claim 1, characterized in that: Step 1, specifically, Step 1.1: Obtain image data and corresponding text data for cross-modal image-text retrieval; Step 1.2: Encode the image data by dividing the input image into multiple non-overlapping image patches. Input these patches into an image encoder for feature extraction to obtain the global feature vector of the image. and image patch feature set ,in, Represents a C-dimensional real vector space. This represents the feature vector of the first to the Mth graphic patches. The set contains M patches, each patch being represented by a C-dimensional vector. Step 1.3: Encode the text data, converting it into a text sequence composed of words, and input the text sequence into a text encoder for feature extraction to obtain sentence-level feature vectors. and word-level feature set Where s represents the sentence-level feature vector; Represents a C-dimensional real vector space; This represents the word-level feature vector from the 1st to the Nth word. The set contains N words, each represented by a C-dimensional vector.

3. The image and text retrieval method based on multi-granularity alignment as described in claim 1, characterized in that: Step 2, specifically, Step 2.1, for image samples Sentence Sample Cosine similarity is calculated on the global feature vector and sentence-level feature vector of the image obtained in step 1: ， Where sim represents the similarity function, which measures the semantic matching degree between images and text; cos represents the cosine similarity function, which calculates the directional similarity between two feature vectors in the feature space; g represents the global granularity, which distinguishes the similarity calculation at different granularity levels. This indicates that the image encoder processed the first... Image samples The extracted global feature vector of the image; This indicates that the text encoder is used for the first... Sentence samples Extracted sentence-level feature vectors; Step 2.2: Organize the multiple image-text coarse-grained similarities obtained in Step 2.1 to obtain coarse-grained similarity data of the overall semantic matching relationship between the image and the overall semantic matching relationship between the text.

4. The image and text retrieval method based on multi-granularity alignment as described in claim 2, characterized in that: Step 3, specifically, Step 3.1: Calculate the similarity between the image patch feature set and the word-level feature set obtained in Step 1. Patch - word similarity matrix; Step 3.2: Based on the patch-word similarity matrix, analyze the similarity distribution between each image patch and text word, determine the image patch features with a semantic correlation higher than the set value with the text word level, and use the image patch features as local matching features.

5. The image and text retrieval method based on multi-granularity alignment as described in claim 4, characterized in that: Step 4, specifically, Step 4.1: Based on the patch-word similarity matrix obtained in Step 3, and combined with the local matching features obtained in Step 3.2, the similarity values ​​corresponding to the local matching features in the patch-word similarity matrix are filtered and retained to obtain the fine-grained similarity relationship between each image patch and each text word, which serves as the input data for subsequent bidirectional interactive aggregation processing. Step 4.2: In the matching direction of the image relative to the text, for each word in the text word-level feature set, based on the fine-grained similarity relationship obtained in Step 4.1, calculate the similarity distribution between the word and each image patch in the image, and obtain the first weight distribution of the word on different image patches through normalization; use the first weight distribution to perform weighted aggregation of the corresponding patch-word similarity to obtain the first matching score of each word on the whole image; perform weighted summation of the first matching scores corresponding to all words to obtain the fine-grained similarity result of the image relative to the text in the first direction. Step 4.3: In the matching direction of text relative to image, for each image patch in the image patch feature set, based on the fine-grained similarity relationship obtained in Step 4.1, calculate the similarity distribution between the image patch and each word in the text, and obtain the second weight distribution of the image patch on different words through normalization; use the second weight distribution to perform weighted aggregation of the corresponding patch-word similarity to obtain the second matching score of each image patch on the entire text; perform weighted summation of the second matching scores corresponding to all image patches to obtain the second-direction fine-grained similarity result of text relative to image; Step 4.4: The fine-grained similarity result of the image relative to the text in the first direction obtained in Step 4.2 is fused with the fine-grained similarity result of the text relative to the image in the second direction obtained in Step 4.3, as follows: ， in, This represents the fine-grained fusion result obtained by comprehensively considering the bidirectional matching relationship between the i-th image sample and the j-th text sample at a fine-grained level; This represents the fine-grained similarity result between the i-th image sample and the j-th text sample in the matching direction of the image relative to the text; This represents the fine-grained similarity result between the i-th image sample and the j-th text sample in the matching direction of the text relative to the image; Step 4.5: Combining the coarse-grained similarity data obtained in Step 2, the fine-grained fusion result obtained in Step 4.4 is constrained using a global semantic constraint method to obtain a multi-granularity image-text similarity that integrates global and local information, and the matching result with the highest similarity is taken as the image-text retrieval result.

6. The image and text retrieval method based on multi-granularity alignment as described in any one of claims 1-5, characterized in that: In step 5, an alignment loss function is constructed to constrain the text-image matching relationship, specifically as follows: Based on multi-granularity image-text similarity data, an alignment loss function is constructed. This function considers the similarity constraints in both the image-to-text direction and the text-to-image direction, aiming to narrow the similarity of matching image-text pairs and widen the similarity of mismatched pairs. The alignment loss function L is: ， in, and Let represent the contrast loss from image to text and from text to image, respectively, and define them using the following formulas: ， ， in, Indicates the similarity of matched pairs, off-diagonal. and Indicates a mismatched image / text pair; This represents the total number of samples within a batch. Indicates the index of the sample currently being processed. Indicates the index of the comparison sample for the summation calculation.

Images