Multi-modal image-text retrieval method fusing fine-grained local semantics and global semantics

Abstract

The invention discloses a multi-modal image-text retrieval method fusing fine-grained local semantics and global semantics, which comprises the following steps of: firstly, extracting original features of an image and a text, and performing regional relation reasoning to obtain relation-enhanced local features; and then semantic interaction is carried out on different samples in the same mode by utilizing an attention mechanism, so that the incidence relation between the samples in each mode is fully learned, and semantic enhanced image text embedding is obtained. And finally, training the whole model by adopting a triad loss function improved by triangular constraint. According to the cross-modal image-text retrieval method, semantic similarity and difference are fully mined, the capability of distinguishing semantic fuzzy samples by the model is enhanced, and the problems that in existing cross-modal image-text retrieval, the recognition accuracy of subtle differences in difficult-to-distinguish samples with subtle differences or semantic fuzzy is low and the like are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal image-text retrieval, and more specifically, relates to a multimodal image-text retrieval method that integrates fine-grained local semantics and global semantics. Background Art

[0002] Image-text retrieval is a crucial task at the intersection of computer vision and natural language processing. It aims to enable users to find corresponding images based on text queries or obtain appropriate text descriptions based on image queries through visual and language understanding. Its goal is to achieve cross-modal semantic alignment between images and text, which is crucial for achieving a deep fusion of vision and language. However, due to the significant semantic differences between images and text, existing methods still do not fully utilize the semantic information in both images and text, and further optimization is urgently needed.

[0003] Image-text retrieval methods typically begin by extracting visual and textual features through modal embedding. They then leverage modal interaction to learn similar features, establish associations and alignments between the two modalities, and exploit semantic relationships across the data. Finally, similarity calculation uses specific criteria to quantify the semantic relevance between the different modal data, achieving effective alignment between the modalities. Similarity calculation criteria typically employ a triplet loss function to distinguish semantically similar from dissimilar samples, but these methods lack fine-grained discrimination and are inadequate for handling difficult samples. For example, for a topic like "surfing," there are subtle differences between different actions, such as "man / woman" and "holding a surfboard / squatting on a surfboard / standing on a surfboard." Existing models may not correctly identify difficult samples with subtle or ambiguous differences, thus compromising modal alignment. Furthermore, the triplet loss suffers from difficulties in finding an appropriate margin parameter and is sensitive to scale variations. Improper margin parameter selection can easily lead to poor embedding learning performance.

[0004] In the cross-modal image-text retrieval method with multi-granularity feature fusion in CN115033670B, the technical problem that the existing cross-modal retrieval method cannot fully extract the information of modal data and cannot perform fine-grained information interaction between cross-modal data is addressed. The disclosed steps include: establishing an image-text retrieval dataset; image-text feature extraction; constructing an attention-based fine-grained cross-modal interaction network and a multi-granularity feature fusion network; using a joint loss function to train the interaction network and the fusion network to complete bidirectional retrieval of images and text. In this patent, by constructing a multi-granularity cross-modal interaction network, not only is information interaction between different modalities performed at a fine-grained level, but also by constructing a multi-granularity feature fusion network, the features of different granularities under the two modalities are fused, so that the image and text embeddings learned by the model can contain global semantic concepts and local semantic information, the information expressed is more comprehensive and accurate, and the data representations of different modalities are closer in the feature space, thereby improving the retrieval accuracy. Although this method improves the accuracy of information to a certain extent, it has obvious deficiencies in utilizing subtle differences in samples. Especially in unbalanced datasets, subtle differences may have a significant impact on classification tasks, resulting in a decrease in retrieval accuracy.

[0005] In addition, the use of a joint loss function in this patent also speeds up model training, allowing the model of this patent to maintain retrieval performance while having higher retrieval efficiency than the cross-modal similarity measurement method. However, the use of a combination of triple loss, corner loss, and text generation loss still causes model redundancy and low learning effect on subtle differences in difficult-to-distinguish samples. Summary of the Invention

[0006] In order to overcome the problems of low recognition accuracy of subtle differences in difficult samples with subtle differences or ambiguous semantics and insufficient learning in existing cross-modal image-text retrieval, the present invention provides a multimodal image-text retrieval method that integrates fine-grained local semantics and global semantics.

[0007] The present invention is achieved through the following technical solutions:

[0008] A multimodal image-text retrieval method integrating fine-grained local semantics and global semantics, comprising the following steps:

[0009] S1. Feature extraction;

[0010] Extract image region features V and text features C respectively;

[0011] S2. Regional relationship reasoning;

[0012] S21. Image region relationship reasoning;

[0013] First, a relationship matrix is ​​constructed based on the similarity between image regions. The image region relationship graph is inferred through the Graph Attention Network (GAT) to generate relationship-enhanced local image features. Then, the original and enhanced features are aggregated to obtain the global image embedding features.

[0014] S22. Textual relationship reasoning;

[0015] Build a semantic relationship graph between words in the text, use the self-attention layer to obtain relationship-enhanced word features, aggregate the original and enhanced features, and obtain the global text embedding features;

[0016] S3.Semantic relationship enhancement;

[0017] S31. Given N sets of image-text pairs Calculate the similarity between the same samples and obtain the embedding relationship matrix:

[0018]

[0019] S32. Given N groups of image-text pairs For samples within the same modality, the correlation weight matrix is ​​calculated using the global embedding feature:

[0020]

[0021] S33. Using the embedding relationship matrix A as the attention mask matrix of the attention module, and the correlation weight matrix X as the additional attention weight matrix, combined with the multi-head attention mechanism, the semantic interaction of the intra-modal samples is performed to obtain the enhanced features. and

[0022] The formula of the multi-head attention mechanism is expressed as:

[0023]

[0024] in, is the original attention weight matrix, λ is the parameter that balances X and the original attention weight matrix;

[0025] The relationship matrix A defines the locations that can “follow” each other, while the correlation weight matrix X is embedded by introducing

[0026] The λ term strengthens the semantic relationship between modalities, allowing the model to more accurately focus on subtle features that may have an impact on classification or recognition.

[0027] S4. Based on the enhanced features and The loss function is obtained by combining the triple ranking loss with the triangle constraint, and the similarity metric training is performed using this loss function;

[0028] The loss function is expressed as:

[0029] L(v,c)=L triplet (v,c)+ωL angular (v,c).

[0030] Furthermore, the image region feature extraction step includes:

[0031] Extract features from the salient areas in the image and map the features of each area to fixed-dimensional image area features through the fully connected layer, expressed as:

[0032]

[0033] where v i Encode the characteristics of a salient region in the image.

[0034] Furthermore, the Faster-RCNN model with Resnet101 as the backbone is used to extract image region features.

[0035] Furthermore, the text feature extraction step includes:

[0036] Extract the features of each word in the text and add a fully connected layer to map the text features to the same dimension as the image features. The text features are represented as:

[0037]

[0038] Furthermore, the BERT model in the field of natural language processing is used to extract text features.

[0039] Furthermore, the image region relationship reasoning step of S21 specifically includes:

[0040] S211. Measure the pairwise similarity between image regions and construct the relationship matrix R:

[0041] R(v i ,v j )=ω(v i ) T μ(v j )

[0042] Among them, ω and μ represent the weight matrix W ω and W μ Feature extraction function for linear transformation;

[0043] S212. Use image region feature V = {v1, v2, v3, ..., v m} is used as the node V of the graph, and the relationship matrix R is used as the edge E of the graph to construct the relationship graph GI =(V,E);

[0044] S213. Using Graph Attention Networks to Reason over Fully Connected Graphs to Capture Semantic Relations and Learn Relation-Enhanced Image Region Features

[0045] S214. Aggregate the original image region feature V and the enhanced image region feature V* through maximum pooling and average pooling to obtain the global visual embedding Expressed as:

[0046] v=η·MaxPool(V)+(1-η)·AvgPool(V * )

[0047] The parameter η is used to control the ratio of the two representations;

[0048] Furthermore, the text region relationship reasoning step of S22 specifically includes:

[0049] S221. Constructing a semantic relationship graph G between words in the text T =(V,E),G T =(V,E) where the node V represents the word feature C = {c1,c2,c3,...,c n}, edges represent the semantic relationship between words;

[0050] S222. Using self-attention layer to obtain relation-enhanced word features C * ={c1 * ,c2 * ,c3 * ,...,c n *};

[0051] S223. Use maximum pooling and average pooling to aggregate these word features to obtain global text embedding Expressed as:

[0052] u=η·MaxPool(C)+(1-η)·AvgPool(C * ).

[0053] Furthermore, ω(v i )=W ω v i and μ(v j )=W μ v j are two embedding features, ω(v i )=W ω v i ,μ(v j )=Wμ v j , weight parameter W ω and W μ Learning through backpropagation.

[0054] Furthermore, the triple ranking loss is expressed as:

[0055]

[0056] Where α is the margin parameter, [x] + = max(x, 0), S(·) is the similarity function in the joint embedding space, (v, c) is a pair of positive samples, and is a set of negative sample pairs, and the hard-to-classify samples are and given.

[0057] Furthermore, the triangle constraint is expressed as:

[0058]

[0059] Where α is the parameter constraining the triangular loss angle, f(a,p,n)=4tan 2 α(a+p)n T -2(1+tan 2 α)ap T ,a, p and n represent image or text embedding respectively, and It is a hard sample in the triangle loss.

[0060] Compared with the prior art, the beneficial effects are:

[0061] Before information interaction, the present invention first generates a relationship matrix and a weight matrix based on the similarity of different samples of the same modality, replaces the mask matrix in the multi-head attention with the relationship matrix, and integrates the weight matrix into the multi-head attention as an additional weight, fully exploring the semantic similarities and differences, extracting additional weights from the global features, and inputting them into the attention mechanism to learn more subtle semantic information, thereby enhancing the model's ability to distinguish semantically ambiguous samples.

[0062] Compared with the existing multi-head attention mechanism, which calculates the correlation between samples to a certain extent through a combination of queries, keys and values, it focuses on information processing within the modality and parallel attention to different features. The present invention constructs a relationship interaction mechanism by using the embedding relationship matrix A and the correlation weight matrix X, which can capture the small but critical relationships between different samples in the embedding space. The relationship matrix can quantify the similarities between different modalities, while the weight matrix determines the degree of "attention" to these similarities. When these matrices are gradually adjusted during the training process, they can help the model identify subtle differences that may be overlooked in traditional methods, thereby producing more discriminative embedding representations. Mainly manifested in:

[0063] (1) More fine-grained feature capture: subtle difference samples can be enhanced, especially the subtle differences between modal interactions and intramodal interactions can be better characterized.

[0064] (2) Improve the generalization ability of the model: By strengthening relational modeling, the model can handle more complex and heterogeneous input data, thereby improving its adaptability to unknown data.

[0065] (3) Enhanced sensitivity to minority class samples: The model can better focus on minority class samples in imbalanced datasets, where subtle differences may have a significant impact on the classification task.

[0066] This paper combines triplet loss and corner loss. For the subtle difference information mined by the feature fusion part, the corner loss is used to constrain the difficult-to-distinguish samples in a targeted manner, further allowing the model to learn the subtle differences in the difficult-to-distinguish samples, achieving excellent performance and no model redundancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a framework diagram of the multimodal image-text retrieval method that integrates fine-grained local semantics and global semantics as described in Example 1;

[0068] Figure 2 Schematic diagram of loss gradients in Example 1, where a is a schematic diagram of triplet loss and its gradient, and b is a schematic diagram of triangular loss and its gradient;

[0069] Figure 3 This is a comparison diagram of examples of the present invention and the VSRN retrieval method in Example 2. DETAILED DESCRIPTION

[0070] The present invention will be further explained and illustrated below with reference to the examples, but the specific examples do not limit the present invention in any form. Unless otherwise specified, the methods and equipment used in the examples are conventional methods and equipment in the art, and the raw materials used are all conventional commercially available raw materials.

[0071] Example 1

[0072] This embodiment provides a multimodal image-text retrieval method that integrates fine-grained local semantics and global semantics, including the following steps:

[0073] S1. Feature extraction;

[0074] S11. Use the Faster-RCNN model with Resnet101 as the backbone to extract features from m salient regions of an image, and use the fully connected layer to map each region into a d-dimensional local feature. The resulting image feature is represented as where v i Encode the characteristics of a salient region in the image.

[0075] S12. Use the BERT model from the natural language processing (NLP) field to extract text features and add a fully connected layer to ensure that the text features have the same size as the image. The resulting text features can be expressed as

[0076] S2. Regional relationship reasoning;

[0077] S21. Image region relationship reasoning;

[0078] S211. Measure the pairwise similarity between image regions and construct the relationship matrix R:

[0079] R(v i ,v j )=ω(v i ) T μ(v j )

[0080] Among them, ω and μ represent the weight matrix W ω and W μ The feature extraction function for linear transformation, ω(v i )=W ω v i and μ(v j )=W μ v j are two embedded features, and the weight parameter W ω and W μ Learning through backpropagation;

[0081] S212. Use image region feature V = {v1, v2, v3, ..., v m} is used as the node V of the graph, and the relationship matrix R is used as the edge E of the graph to construct the relationship graph G I =(V,E);

[0082] S213. Using Graph Attention Networks to Reason over Fully Connected Graphs to Capture Semantic Relations and Learn Relation-Enhanced Image Region Features

[0083] S214. Aggregate the original image region feature V and the enhanced image region feature V* through maximum pooling and average pooling to obtain the global visual embedding feature Expressed as:

[0084] v=η·MaxPool(V)+(1-η)·AvgPool(V * )

[0085] The parameter η is used to control the ratio of the two representations;

[0086] S22. Text region relationship reasoning;

[0087] S221. Constructing a semantic relationship graph G between words in the text T =(V,E),G T =(V,E) where the node V represents the word feature C = {c1,c2,c3,...,c n}, edges represent the semantic relationship between words;

[0088] S222. Using self-attention layer to obtain relation-enhanced word features C * ={c1 * ,c2 * ,c3 * ,...,c n *};

[0089] S223. Use maximum pooling and average pooling to aggregate word features to obtain global text embedding features Expressed as:

[0090] u=η·MaxPool(C)+(1-η)·AvgPool(C * ).

[0091] S3.Semantic relationship enhancement;

[0092] S31. Given N sets of image-text pairs Calculate the similarity between the same samples and obtain the embedding relationship matrix:

[0093]

[0094] S32. Given N groups of image-text pairs For samples within the same modality, the correlation weight matrix is ​​calculated using global embedding:

[0095]

[0096] where σ is a positive scalar that controls the correlation value;

[0097] S33. After obtaining the embedding relationship matrix and weight matrix, a relational interaction mechanism is used to capture semantic relationships. The visual embedding and text embedding are separately input into the multi-head attention module to achieve multimodal relational interaction. This accurately captures the semantic relationship between image and text, further strengthening the deep interaction between different modalities and enabling more effective extraction of cross-modal semantic information, thereby improving the model's performance and accuracy in cross-modal tasks. The query (Q) and key-value pairs (KV) come from two different modalities.

[0098] The embedding relationship matrix A is used as the attention mask matrix of the attention module, and the correlation weight matrix X is used as an additional attention weight matrix for explicit relationship modeling. λ is used to balance X with the original attention weight matrix in the multi-head attention. The matrix is ​​composed of the attention weights formed by the key-value pairs, and the formula for the multi-head attention mechanism is expressed as:

[0099]

[0100] Perform semantic interaction on samples within the modality to obtain features after enhancing the relationship between the two modalities and

[0101] S4. Combine the triple ranking loss with the triangle constraint to obtain a loss function, and use this loss function to perform similarity metric training;

[0102] S41. Image and text features after relationship enhancement and The matching is performed using triple ranking loss based on hard samples, and the loss is defined as:

[0103]

[0104] Where α is the margin parameter, [x] + =max(x,0). S(·) is the similarity function in the joint embedding space, (v,c) is a pair of positive samples, and is a set of negative sample pairs, and the hard-to-classify samples are and given.

[0105] S42. The gradient direction of negative and positive samples in the triple ranking loss will cause the distance between them to become closer, so the triangular loss is introduced:

[0106]

[0107] Where α is the parameter constraining the triangular loss angle, f(a,p,n)=4tan 2 α(a+p)n T -2(1+tan 2 α)ap T ,a, p and n represent image or text embedding respectively. and It is a hard sample in the triangle loss.

[0108] By constructing the inner angle constraint of a triangle with the distance between the anchor point and the positive and negative samples as its sides, we provide an additional constraint source while making the gradient direction of the negative sample points move away from the positive sample points and the anchor point. This avoids the situation in the triple ranking group loss where the negative sample points are pushed away from the anchor point but closer to the positive sample points, thus achieving better alignment.

[0109] S43. Combine the triplet ranking loss and the triangle loss to obtain the final loss function:

[0110] L(v,c)=L triplet (v,c)+ωL angular (v,c)

[0111] The weight ω is used to control the importance of triangular loss.

[0112] Example 2

[0113] In order to verify the effectiveness of the method proposed in Example 1, this example uses two public datasets to evaluate the model, MS-COCO and Flickr30K. MS-COCO contains 123,287 images, each with five corresponding text descriptions. 113,287 images are used for training, 5,000 images are used for verification, and the remaining 5,000 images are used for testing. The test results of MS-COCO consist of the average results of the five-fold cross-validation of 1,000 test samples (COCO 5-fold 1k test) and the full 5,000 test samples (COCO 5k test). Flickr30K contains 31,783 images, each with five text descriptions. 1,014 images are used for verification, 1,000 images are used for testing, and 29,000 images are used for training.

[0114] Text-image retrieval is typically evaluated using the Recall@K (K=1, 5, 10) metric, denoted as R@1, R@5, and R@10. Recall@K represents the percentage of ground truth found in the top K retrieved items, with higher values ​​indicating higher retrieval accuracy. The sum of the three image retrieval metrics and the sum of the three text retrieval metrics are calculated and combined into an overall evaluation metric for text-image retrieval, called rSum:

[0115] rSum=(R@1+R@5+R@10) image

[0116] +(R@1+R@5+R@10) text

[0117] In terms of experimental settings, all experiments were implemented on NVIDIA Tesla T4 using the PyTorch framework. During training, the Adam optimizer was used with an initial learning rate of 1e -5 , decaying by a factor of 0.1 every 10 iterations. The batch sizes for Flickr30k and MS-COCO were set to 128 and 256, respectively. Pre-extracted image region features were used for images, and the image and text features were transformed into d dimensions, with d = 1024. The hyperparameter η, which controls the pooling ratios of the two, was 0.8. For semantic relationship reinforcement learning, the quantile θ% was set to 50% and λ was set to 1.5. The hyperparameters in the loss function were α = 0.2, β = 0.5, and ω = 0.65.

[0118] The experimental results on the MS-COCO dataset are shown in Table 1:

[0119]

[0120] The experimental results on the Flickr30K dataset are shown in Table 2:

[0121]

[0122] Table 1 shows the comparison results of the proposed method and the advanced methods in recent years on the MS-COCO dataset. In the COCO 5K test set, the EISIN model achieved the highest performance in R@K in three indicators. In terms of rSum, it is better than HREM. [3] and VSRN++[ 31] Although the performance of the EISIN model in image retrieval is similar to that of previous methods, it has significant improvements in text retrieval, which are 1% and 1.5% higher than HREM. [3] and VSRN++

[31] The EISIN model also achieves significant performance improvements on the COCO5-fold 1K test set.

[0123] Table 2 shows the comparison results of the proposed method and the advanced methods in recent years on the Flickr30K dataset. In the Flickr30K test set, the EISIN model has four indicators that achieve the highest performance in R@K, rSum and VSRN++[ 31] Compared with the previous report, it has also increased by about 2.3%.

[0124] In addition, according to Figure 3 The two examples of image-text retrieval methods EISIN and VSRN are shown in the figure. The green mark indicates a correct retrieval and the red mark indicates an incorrect retrieval. It can be seen that EISIN is more accurate than VSRN. [2] Have better image and text retrieval capabilities.

[0125] In this embodiment, VSE++

[11] Improving visual-semantic embeddings with hardnegatives[C] / / Proceedings of the 2018British Machine Vision Conference(BMVC).Durham:BMVC,2018:344;SCAN

[16] Stacked cross attention for image-textmatching[C] / / Proceedings of the European conference on computer vision(ECCV).2018:201-216; VSRN [2] For Visual semantic reasoning for image-text matching[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2019:4654-4662; IMRAM

[27] Iterative matching with recurrent attention memory for cross-modal image-text retrieval[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020:12655-12663;CAAN

[28] Context-aware attention network for image-text retrieval[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020:3536-3545;CGMN

[29] Cross-modal graph matching network for image-text retrieval[J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2022, 18(4):1-23;NAAF

[30] Negative-aware attention framework for image-text matching[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022:15661-15670;VSRN++

[31] Image-text embedding learning via visual and textual semantic reasoning[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 45(1):641-656;HREM [3]This is the method described in Learning semantic relationship among instances for image-text matching[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:15159-15168.

[0126] Example 3

[0127] In this example, we conducted ablation experiments on the MS-COCO5K dataset using the main components of the method described in Example 1 and studied their impact on model performance. The experimental results are shown in Table 3:

[0128] Table 3

[0129]

[0130] (1) Effectiveness of Triangular Loss: We examine the performance of the model after removing the Angular Loss (AL). The results are shown in Table 3. Removing the Angular Loss results in an overall decrease of approximately 2.2% in text retrieval and a decrease of approximately 1.0% in rSum. This indicates that the Angular Loss indeed optimizes the gradient direction, moving the gradient direction of negative sample points away from the positive sample points and anchor points, thereby achieving better retrieval results. Compared to text retrieval, the effect of Angular Loss on image retrieval is less significant. This may be because the distribution of features of different image classes in the embedding space is quite different, thus having a greater impact on retrieval.

[0131] (2) Effectiveness of Semantic Relationship Enhanced Learning: The Semantic Relationship Enhanced Learning (SEL) module was removed to study the effect of the model that does not learn different information within the same modality. The results are shown in Table 3. All six evaluation indicators and rSum have decreased significantly, which proves that EISIN enables the model to accurately learn the semantic information between images and texts, and achieve better retrieval results by using more sufficient semantic information, which improves all indicators, proving the effectiveness of the method.

[0132] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A multimodal image-text retrieval method integrating fine-grained local semantics and global semantics, characterized in that: include: S1. Feature extraction; Extract image region features V and text features C respectively; S2. Regional relationship reasoning; S21. Image region relationship reasoning; First, a relationship matrix is ​​constructed based on the similarity between image regions. The image region relationship graph is inferred through a graph attention network to generate relationship-enhanced local image features. Then, the original and enhanced features are aggregated to obtain the global image embedding features. S22. Textual relationship reasoning; Build a semantic relationship graph between words in the text, use the self-attention layer to obtain relationship-enhanced word features, aggregate the original and enhanced features, and obtain the global text embedding features; S3.Semantic relationship enhancement; S31. Given N sets of image-text pairs Calculate the similarity between the same samples and obtain the embedding relationship matrix: S32. Given N groups of image-text pairs For samples within the same modality, the correlation weight matrix is ​​calculated using global embedding: S33. Using the embedding relationship matrix A as the attention mask matrix of the attention module, and the correlation weight matrix X as the additional attention weight matrix, combined with the multi-head attention mechanism, the semantic interaction of the intra-modal samples is performed to obtain the enhanced features. and The formula of the multi-head attention mechanism is expressed as in, is the original attention weight matrix, λ is the parameter that balances X and the original attention weight matrix; S4. Based on the enhanced features and The loss function is obtained by combining the triple ranking loss with the triangle constraint, and the similarity metric training is performed using this loss function; The loss function is expressed as: L(v,c)=L triplet (v,c)+ωL angular (v,c)。 2. The multimodal image-text retrieval method integrating fine-grained local semantics and global semantics according to claim 1 is characterized in that: The image region feature extraction steps include: Extract features from the salient areas in the image and map the features of each area to fixed-dimensional image area features through the fully connected layer, expressed as: V={v1,v2,v3,...,v m }, where v i Encode the characteristics of a salient region in the image.

3. The multimodal image-text retrieval method integrating fine-grained local semantics and global semantics according to claim 2 is characterized in that: The Faster-RCNN model with Resnet101 as the backbone is used to extract image region features.

4. The multimodal image-text retrieval method integrating fine-grained local semantics and global semantics according to claim 1 is characterized in that: The text feature extraction steps include: Extract the features of each word in the text and add a fully connected layer to map the text features to the same dimension as the image features. The text features are represented as: C={c1,c2,c3,...,c n }, 5. The multimodal image-text retrieval method integrating fine-grained local semantics and global semantics according to claim 4 is characterized in that: The BERT model in the field of natural language processing is used to extract text features.

6. The multimodal image-text retrieval method integrating fine-grained local semantics and global semantics according to claim 1 is characterized in that: The image region relationship reasoning steps of S21 specifically include: S211. Measure the pairwise similarity between image regions and construct the relationship matrix R: R(v i ,v j )=ω(v i ) T μ(v j ) Among them, ω and μ represent the weight matrix W ω and W μ Feature extraction function for linear transformation; S212. Use image region feature V = {v1, v2, v3, ..., v m } is used as the node V of the graph, and the relationship matrix R is used as the edge E of the graph to construct the relationship graph G I =(V,E); S213. Use graph attention network fully connected graph to reason about semantic relations and learn relation-enhanced image region features V*={v1 * ,v2 * ,v3 * ,...,v m * }, S214. Aggregate the original image region feature V and the enhanced image region feature V* through maximum pooling and average pooling to obtain the global visual embedding feature Expressed as: v=η·MaxPool(V)+(1-η)·AvgPool(V * ) The parameter η is used to control the ratio of the two representations.

7. The multimodal image-text retrieval method integrating fine-grained local semantics and global semantics according to claim 1 is characterized in that: The text region relationship reasoning steps of S22 specifically include: S221. Constructing a semantic relationship graph G between words in the text T =(V,E),G T =(V,E) where the node V represents the text feature C = {c1,c2,c3,...,c n }, edges represent the semantic relationship between words; S222. Using self-attention layer to obtain relation-enhanced text features C * ={c1 * ,c2 * ,c3 * ,...,c n * }; S223. Use maximum pooling and average pooling to aggregate these word features to obtain global text embedding features Expressed as: u=η·MaxPool(C)+(1-η)·AvgPool(C * ).

8. The multimodal image-text retrieval method integrating fine-grained local semantics and global semantics according to claim 1 is characterized in that: ω(v i )=W ω v i and μ(v j )=W μ v j are two embedding features, ω(v i )=W ω v i ,μ(v j )=W μ v j , weight parameter W ω and W μ Learning through backpropagation.

9. The multimodal image-text retrieval method integrating fine-grained local semantics and global semantics according to claim 1, characterized in that: The triplet ranking loss is expressed as: Where α is the margin parameter, [x] + = max(x, 0), S(·) is the similarity function in the joint embedding space, (v, c) is a pair of positive samples, and is a set of negative sample pairs, and the hard-to-classify samples are and given.

10. The multimodal image-text retrieval method integrating fine-grained local semantics and global semantics according to claim 1, characterized in that: The triangle constraint is expressed as: Where α is the parameter constraining the triangular loss angle, f(a,p,n)=4tan 2 α(a+p)n T -2(1+tan 2 α)ap T ,a, p and n represent image or text embedding respectively, and It is a hard sample in the triangle loss.

Images