Image-Text Matching Method and System Based on the Fusion of Attention and Similarity under Graph Inference
Through the method of integrating attention and similarity under graph reasoning, the problem of inaccurate attention mechanism and ignoring deep semantics in graph-text matching is solved, and a more accurate and easy-to-explain graph-text matching result is achieved.
Patent Information
- Application Number
- CN202311587133.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-24
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2043-11-24
AI Technical Summary
In the existing graphic matching methods, attention mechanisms are susceptible to noise, resulting in inaccurate weight allocation, similarity measurement ignores deep semantic relationships, and imbalance in the number of images and texts affects the accuracy and fairness of the matching results.
The attention and similarity fusion method based on graph reasoning is adopted to improve matching accuracy through feature extraction, similarity matrix construction, self-attention graph reasoning and bipolar attention control, combining local and global features.
It enhances the discernibility and robustness of graphic matching, and the semantic expression is more complete and unified, which improves the accuracy and ease of interpretation of the matching.
Smart Images
Figure CN117521013B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical fields of natural language processing and computer vision. Specifically, it relates to a method and system for image-text matching that fuses attention and similarity based on graph reasoning. Background Art
[0002] The statements in this section merely provide background technical information related to the present disclosure and do not necessarily constitute prior art.
[0003] Image-text matching refers to the process of realizing semantic relevance modeling and matching between images and texts through computer vision and natural language processing technologies. Among numerous modal data, visual and text-form data dominate, and the cross-modal matching task between them has gradually become a hot topic.
[0004] The inventors found in their research that in existing methods using the attention mechanism in image-text matching, due to the fact that the attention mechanism may be affected by input noise or interference, resulting in inaccurate weight allocation. If there is noise or incorrect information in the input data, the attention mechanism may wrongly focus on these useless parts, affecting the performance of the model.
[0005] In existing research focusing on similarity, some have achieved more accurate and semantically rich image-text matching by learning similarity embeddings between images and texts. Some have introduced a text-to-image synthesis method based on graph convolutional networks, which has achieved accurate and semantically consistent image-text matching and synthesis by learning the similarity between texts and images. However, in the above existing methods using similarity, since the similarity metric usually only considers local or surface similarity and ignores the deeper semantic relationship between images and texts, it leads to inaccuracy of the matching result and lack of semantic consistency. At the same time, the number of images and texts may be unbalanced, which may cause the similarity metric to be overly biased towards a certain modality, thus affecting the fairness and accuracy of the matching result. Summary of the Invention
[0006] To solve the above problems, the present disclosure proposes a method and system for image-text matching that fuses attention and similarity based on graph reasoning, which can take into account the advantages of both the attention mechanism and the similarity metric and improve the accuracy of matching.
[0007] To achieve the above object, the present disclosure adopts the following technical solutions:
[0008] One or more embodiments provide a method for image-text matching that fuses attention and similarity based on graph reasoning, including the following steps:
[0009] Extract features from the matched text and images respectively to obtain word-level features, sentence-level features, image region features, and image global features;
[0010] Calculate the local similarity by computing the similarity between the image region features and the word-level features, calculate the global similarity by computing the similarity between the image global features and the sentence-level features, and perform alignment and fusion to obtain a similarity matrix;
[0011] Based on the similarity matrix, represent the image and text data as a graph structure, and use self-attention graph reasoning to obtain a similarity matrix with implicit relationships;
[0012] Use the soft threshold control of bipolar attention, and adopt bipolar attention to focus on the unmatched and matched segments in the image and text pairs to obtain the similarity of metric word segments and the weighted similarity of word segments;
[0013] Based on the obtained similarities, perform matching calculations to obtain the image and text matching results.
[0014] One or more embodiments provide a graphic-text matching system based on the fusion of attention and similarity under graph reasoning, including:
[0015] Feature extraction module: configured to extract features from the text and images to be matched respectively to obtain word-level features, sentence-level features, image region features, and image global features;
[0016] Similarity calculation module: configured to calculate the local similarity by computing the similarity between the image region features and the word-level features, calculate the global similarity by computing the similarity between the image global features and the sentence-level features, and perform alignment and fusion to obtain a similarity matrix;
[0017] Self-attention graph reasoning module: configured to represent the image and text data as a graph structure based on the similarity matrix, and use self-attention graph reasoning to obtain a similarity matrix with implicit relationships;
[0018] Bipolar attention module: configured to use the soft threshold control of bipolar attention, and adopt bipolar attention to focus on the unmatched and matched segments in the image and text pairs to obtain the similarity of metric word segments and the weighted similarity of word segments;
[0019] Fusion matching module: configured to perform matching calculations based on the obtained similarities to obtain the image and text matching results.
[0020] An electronic device includes a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps in the above-mentioned graphic-text matching method based on the fusion of attention and similarity under graph reasoning are completed.
[0021] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the steps in the above-mentioned image-text matching method based on the fusion of attention and similarity under graph reasoning are completed.
[0022] Compared with the prior art, the present invention has the following beneficial effects:
[0023] The present disclosure can take into account the advantages of both the attention mechanism and the similarity measurement, using the attention mechanism and the threshold to control the matching of image and text segments and the non-matching image and text segments; allowing the positive influence of the matching segments and the negative influence of the non-matching segments to act simultaneously, thereby increasing the discriminability and robustness; and utilizing the similarity fusion of global features and local features to measure the similarity between images and texts in an intuitive way while considering low computational costs, thereby making the semantic representation more complete and unified and improving the accuracy of matching.
[0024] The advantages of the present disclosure and additional advantages will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure but do not constitute a limitation of the present disclosure.
[0026] Figure 1 is a block diagram of an image-text matching system based on attention and similarity fusion under graph reasoning according to embodiment 2 of the present disclosure;
[0027] Figure 2 This is a flowchart of the image-text matching method based on the fusion of attention and similarity under graph reasoning in Example 1 of the present disclosure. DETAILED DESCRIPTION
[0028] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.
[0029] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.
[0030] Note that the terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly dictates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features in the present disclosure may be combined with each other. The embodiments will be described in detail below with reference to the drawings.
[0031] Embodiment 1
[0032] In the technical solutions disclosed in one or more embodiments, as Figures 1 to 2 shown, a text-image matching method based on the fusion of attention and similarity under graph reasoning includes the following steps:
[0033] Step 1: Extract features from the text and image to be matched respectively to obtain word-level features, sentence-level features, image region features, and image global features;
[0034] Step 2: Calculate the similarity between the image region features and the word-level features to obtain the local similarity, calculate the similarity between the image global features and the sentence-level features to obtain the global similarity, and perform alignment and fusion to obtain the similarity matrix S v ;
[0035] Step 3: Represent the image and text data as a graph structure based on the similarity matrix, and use self-attention graph reasoning to obtain the similarity matrix S with implicit relationships v ′ ;
[0036] Step 4: Use the soft threshold control of bipolar attention, and use bipolar attention to focus on the mismatched and matched segments in the image and text pair to obtain the similarity s j of the metric word segment u f , the weighted similarity s j of the word segment u r ;
[0037] Step 5: Perform matching calculations based on the obtained similarities to obtain the image-text matching result;
[0038] Among them, in Step 5, matching calculations are performed based on the obtained similarities. Among them, the similarities include the similarity matrix S with implicit relationships v ′ , the similarity s j of the metric word segment u f , and the weighted similarity s j of the word segment u r ;
[0039] S v ′ 、s f s r In this embodiment, graph inference is introduced to learn the node representation and edge weights on the graph, which can capture the associations, semantic relationships, and context information between images and texts. For the matching and non-matching segments in image-text matching, a soft threshold control using bipolar attention is proposed, which can improve discrimination and robustness, making the semantic representation more complete and unified, and enhancing the interpretability and comprehensibility of the matching process. Fusing and regulating bipolar attention and similarity can increase discrimination and robustness, making the semantic representation more complete and unified, and enhancing the interpretability and comprehensibility of the matching process.
[0040] In step 1, feature extraction is performed, including feature extraction for images and texts respectively.
[0041] (1) Feature extraction for images includes: detecting significant regions in the image to be matched, encoding and transforming the detected regions to obtain image region features and image global features;
[0042] Specifically, to extract image regions with visual semantic expressions, Faster R-CNN is used to detect significant regions in image I, and the detected image region r i is encoded to obtain the visual representation x i ; and x i is transformed into a d-dimensional v i through linear projection, where v i represents the image region, which is the image region feature, and i represents the number of image regions;
[0043] Using the average eigenvalue v ave as a query, the global representation v glo of the entire image I is obtained, which is the image global feature. Specifically, using a bottom-up attention mechanism, the overall image representation v glo is obtained by weighted averaging the attention of each local region in the image: the average feature v ave is used as the basis for the query and is used to calculate the weight of each region, and these weights are then used to weighted average the visual vectors of each region to form the global representation. The specific formula is as follows:
[0044] v i = W v x i + b i
[0045]
[0046] where Wv Denotes the weight, b i Denotes the bias, w i Denotes v ave And v i The attention weight between them.
[0047] (2) Feature extraction of the text, including:
[0048] Extract word-level text semantic information to capture the fine-grained interaction between vision and language.
[0049] First, the words in the text T to be matched are one-hot encoded through a learnable word embedding layer and mapped to a distributed representation.
[0050] Secondly, in order to enhance the text representation through context semantics, bidirectional GRU is used to encode the forward and backward information to obtain enhanced word-level features, and the enhanced word-level features u j Is defined as the average of the hidden states of the forward and backward GRUs of the bidirectional GRU encoding at the j-th step.
[0051] The sentence-level features u of the text glo The acquisition method is basically the same as that of the global features v of the image glo The acquisition method is used. The average feature value is used as a query to obtain the global representation of the entire text, which is the sentence-level feature.
[0052] The hidden states of the forward and backward GRUs of the bidirectional GRU encoding at the j-th step are calculated as follows:
[0053]
[0054]
[0055] Among them, Respectively represent the hidden states of the forward and backward GRUs at the j-th step (as the current step), Is a variant of the recurrent neural network (RNN) for processing sequence data; t j The learnable word embedding layer at the j-th step, Is the hidden state of the previous step, Is the hidden state of the next step.
[0056] In step 2, calculating the image region-word level similarity and the global image feature-sentence similarity and fusing them can strengthen the correlation between the visual background of the region and the semantics of the complete description in the image;
[0057] Among them, the method for calculating and aligning and fusing the local similarity and the global similarity includes the following steps:
[0058] Step 21: Perform an embedding operation on the extracted image features and text features to obtain local image embedding vectors, text embedding vectors, global image embedding vectors, and global text embedding vectors, including the following processes:
[0059] Step 211: Map the image features and text features through an embedding layer to obtain local image embedding vectors and text embedding vectors;
[0060] Step 212: Obtain the global image embedding vector by averaging the local image embedding vectors, and obtain the global text embedding vector by performing global weighted averaging on the text embedding vectors;
[0061] Step 213: Expand the global image embedding vector so that the shape of the expanded global image embedding vector is consistent with the embedding vector of the image region;
[0062] Step 214: Obtain the common feature vector by performing element-wise multiplication on the local image embedding vector and the global image embedding vector;
[0063] Step 215: Obtain the attention weights by performing Softmax normalization on the weights, and obtain the updated global image embedding vector by performing weighted summation of the attention weights and the common feature vector;
[0064] Step 22: Align and construct the local-global embedding vectors of the image and text:
[0065] u j represents the embedding vector of the j-th text. Expand the embedding vector of the j-th text to make its shape consistent with the image embedding vector; Through the SCAN (Stacked Cross Attention Network) algorithm, construct a local-global aligned image context between the text and the image;
[0066] Through the SCAN algorithm, the system attempts to establish a correspondence between the text and the image, so that the local details and the overall structure can be aligned in the two domains. This helps to better understand the relationship between the text description and the image content, and provides more comprehensive information for cross-modal tasks.
[0067] Step 23: Based on the aligned embedding vectors, perform similarity calculation and alignment fusion, including the following steps:
[0068] Step 231: Calculate the local similarity between the image and the text based on the image region features and text context information. The formula is:
[0069]
[0070] where, is the set of learnable first parameter matrices, is the text context, is the local similarity between the image and the text;
[0071] Step 232: According to the global image feature v glo and the global text feature u glo , calculate the global similarity between the image and the text. The formula is:
[0072]
[0073] where s glo is the global similarity between the image and the text, is the set of learnable second parameter matrices, v glo represents the global image feature, and u glo represents the global text feature.
[0074] Step 233: Concatenate the global-local similarity and the local similarity to obtain the similarity matrix S v , and then flatten the similarity matrix into a one-dimensional vector;
[0075]
[0076] where c N represents the matching confidence: showing the degree of consistency of each region from the global perspective of image-text, and filtering out the region-word pairs with inconsistent matches.
[0077] Furthermore, the similarity matrix S obtained by concatenating the global alignment and the region alignment v constructs an inference graph, and uses self-attention graph inference and loops multiple times to obtain the similarity matrix S with implicit relationships v ′ , and by using self-attention graph inference, the association, semantic relationship, and context information between the image and the text can be captured by learning the node representations and edge weights on the graph.
[0078] where the similarity matrix S with implicit relationships v ′ is the enhancement of the similarity matrix S v .
[0079] In this embodiment, by modeling and the inference graph structure, the association and dependency relationships between objects can be analyzed and inferred, and then logical reasoning and relationship reasoning can be performed.
[0080] In step 4, bipolar attention can simultaneously focus on the mismatched and matched segments in the image-text pair, and by using different attention masks, their effects in negative attention and positive attention can be accurately measured.
[0081] In this embodiment, bipolar attention calculation is specifically as follows:
[0082] Step 41: Calculate the semantic relevance scores between all text words and image regions using the formula:
[0083]
[0084] Among them, u j is the embedding vector of the jth text, v i is the embedding vector of the i-th image region, which is also the regional feature of the image.
[0085] u j is the embedding vector of the jth text, which can also be called a word fragment;
[0086] Step 42: Perform negative attention calculation based on the relevance score to calculate the value of each word segment u j With all image regions v i The similarity s j , and propagate within the text modal;
[0087]
[0088] Among them, s ij is the semantic relevance score t between all text words and image regions k is the optimal boundary, u j is the embedding vector of the jth text, which can also be called a word fragment;
[0089] Thus, the negative attention can be calculated as:
[0090] Since semantically similar word fragments have the same matching relationship, the matching degree of each word is propagated intramodally:
[0091]
[0092] in, Represents the j-th word segment u j and the lth word fragment u l The semantic relationship between
[0093] A segment without a matching image region in the text modality is considered to be mismatched. The maximum cross-modal similarity between a segment and all segments in the other modality reflects the degree of mismatch or match. This embodiment calculates the maximum cross-modal similarity between each word segment and all image regions v in the image modality. i The similarity between the two fragments can be accurately identified.
[0094] Step 43: Calculate the positive attention based on the relevance score, aggregate the matching image regions for each query word, and calculate the similarity of the metric word segment u j similarity and the weighted similarity of each word segment u j in the text
[0095] First, focus on the cross-modal shared semantics, that is, aggregate the matching image regions for each query word. Specifically, the method for calculating the multimodal attention weights is as follows:
[0096]
[0097] For the j-th word, the corresponding shared semantics in the image can be aggregated to obtain the weighted image feature as:
[0098]
[0099] Based on this weighted image feature, calculate the similarity between the word segment u j and the weighted image feature , which is the similarity of the metric word segment u j , as follows:
[0100]
[0101] According to the relevance score s ij between the text and the image, calculate the weighted similarity of the word segment u j as follows: as follows:
[0102]
[0103] where means the relevance weight.
[0104] Furthermore, use a feed-forward neural network to match the dimensions of the similarity and bipolar attention.
[0105] Furthermore, fuse the bipolar attention result and the similarity to obtain the final score. The specific fusion method is as follows:
[0106] The similarity of the matching text-image pair is calculated as:
[0107]
[0108] Finally, the similarity of the image-text (U, V) can be jointly determined by the positive and negative effects. The total similarity S(U, V) between the image to be recognized and the text is the weighted sum of the similarities, that is, to obtain the matching result of the text and image, and the calculation formula is:
[0109]
[0110] Furthermore, the positive and negative effects are controlled by a threshold to distinguish the matching fragments and non-matching fragments as much as possible; by adjusting the attention weights and the values of the similarity metric, the attention distribution, similarity range, and sparsity during the matching process can be controlled, thereby improving the accuracy, robustness, and generalization ability of the matching.
[0111] Both the soft threshold and the hard threshold are threshold-based signal processing techniques used to process noise and clutter in signals. Their function is to filter out small fluctuations in the signal and retain large fluctuations for better signal analysis and processing. In this embodiment, the soft threshold is adopted.
[0112] The soft threshold sets to 0 the parts of the signal whose absolute values are less than a preset threshold, and subtracts half of the threshold from the parts greater than the threshold. Unlike the hard threshold, the soft threshold does not completely suppress small fluctuations in the signal but compresses them to better retain the detailed information in the signal. The signal processed by the soft threshold is smoother than that processed by the hard threshold, but there may be some smooth transitions.
[0113] Furthermore, all the scoring results are retained and concatenated, and contrastive loss is used for prediction optimization and parameter optimization to obtain the optimal model. Specifically, the loss function used is the bidirectional triplet ranking loss. Given a true image-text pair (U, V) and all its non-matching pairs (U, V ′ ) and (U ′ , V), the calculation formula of the loss function is:
[0114]
[0115] where γ is the margin hyperparameter, [] + represents an operation method. The value x inside [] is compared with 0. If the value is greater than or equal to zero, then [] + is equal to x, otherwise it is equal to 0.
[0116] Embodiment 2
[0117] Based on Embodiment 1, this embodiment provides a text-image matching system that fuses attention and similarity based on graph reasoning, including:
[0118] Feature extraction module: configured to perform feature extraction on the text and picture to be matched respectively to obtain word-level features, sentence-level features, image region features, and image global features;
[0119] Similarity calculation module: It is configured to calculate the similarity between image regional features and word-level features to obtain local similarity, calculate the similarity between image global features and sentence-level features to obtain global similarity, and perform alignment and fusion to obtain a similarity matrix;
[0120] Self-attention graph reasoning module: It is configured to represent image and text data as a graph structure based on the similarity matrix and use self-attention graph reasoning to obtain a similarity matrix with implicit relationships;
[0121] Bipolar attention module: configured to use soft threshold control of bipolar attention, using bipolar attention to focus on the mismatched and matched segments in the image and text pairs, and obtain the similarity of measured word segments and the weighted similarity of word segments;
[0122] Fusion matching module: configured to perform matching calculation based on the obtained similarities to obtain image and text matching results.
[0123] Among them, the similarity calculation module is divided into a local similarity calculation module and a global similarity calculation module.
[0124] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.
[0125] Example 3
[0126] Based on Example 1, this embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the steps of the image-text matching method based on attention and similarity fusion under graph reasoning described in Example 1 are completed.
[0127] Example 4
[0128] Based on Example 1, this embodiment provides a computer-readable storage medium, characterized in that it is used to store computer instructions. When the computer instructions are executed by the processor, the steps in the image-text matching method based on attention and similarity fusion under graph reasoning described in Example 1 are completed.
[0129] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.
[0130] Although the specific implementation manners of the present disclosure have been described above in conjunction with the accompanying drawings, they are not limitations on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications or deformations that can be made without creative efforts on the basis of the technical solutions of the present disclosure are still within the protection scope of the present disclosure.
Claims
1. A method for image-text matching based on the fusion of attention and similarity under graph reasoning, characterized in that The method includes the following steps: respectively extracting features of the text and the image to be matched to obtain word-level features, sentence-level features, image region features, and image global features; Calculating the similarity between the image region features and the word-level features to obtain a local similarity, calculating the similarity between the image global features and the sentence-level features to obtain a global similarity, and performing alignment and fusion to obtain a similarity matrix; The method for calculating, aligning, and fusing the local similarity and the global similarity includes the following steps: performing an embedding operation on the image features and the text features to obtain a local image embedding vector, a text embedding vector, a global image embedding vector, and a global text embedding vector; obtaining a common feature vector by performing element-wise multiplication on the local image embedding vector and the global image embedding vector; obtaining an attention weight by normalizing the weights, and obtaining an updated global image embedding vector by performing weighted summation of the attention weight and the common feature vector; constructing the obtained embedding vectors for alignment as follows: using the SCAN algorithm to construct a local-global aligned image context between the text and the image; Based on the aligned embedding vectors, perform similarity calculation and alignment fusion; based on the image region features and text context information , calculate the local similarity between the image and the text , and the formula is: ; where is the set learnable first parameter matrix; according to the image global feature and the text global feature , calculate the global similarity between the image and the text , and the formula is: ; where is the set learnable second parameter matrix; Connect the global local similarity and the local similarity to obtain a similarity matrix , and then flatten the similarity matrix into a one-dimensional vector; ; where represents the matching confidence; Representing the image and text data as a graph structure based on the similarity matrix, and using self-attention graph reasoning to obtain a similarity matrix with implicit relationships; Using the soft threshold control of bipolar attention, and using bipolar attention to focus on the unmatched and matched segments in the image-text pair to obtain the similarity of the metric word segments and the weighted similarity of the word segments; Bipolar attention calculation includes the following steps: calculating the semantic correlation scores between all text words and image regions; ; where is the embedding vector of the j-th text, is the embedding vector of the i-th image region and also the regional feature of the image; Negative attention is calculated based on the relevance score, calculating each word segment with all image regions similarity , ; is the optimal boundary, and the negative attention is calculated as: ; and propagated within the text modality; according to the fact that semantically similar word segments have the same matching relationship, the matching degree of each word is propagated within the modality: ； Among them, represents the semantic relationship between the j-th word segment and the l k-th word segment ; According to the correlation score, performing positive attention calculation, aggregating the matching image regions for each query word, and calculating the similarity of the metric word segments and the weighted similarity of each word segment in the text; Performing a matching calculation based on the obtained similarities to obtain an image-text matching result; Fuse the bipolar attention results and similarity to obtain the final score. The specific fusion method is as follows: The similarity of the matched text-image pair is calculated as follows: = + ; represents the similarity of the measure word segment ; represents the weighted similarity of the word segment ; represents the similarity matrix with implicit relationships Total similarity between the image to be recognized and the text Is the weighted sum of the similarities. To obtain the matching result of the image and text, the calculation formula is: .
2. The image-text matching method based on the fusion of attention and similarity under graph reasoning as claimed in claim 1 is characterized in that: The feature extraction of the image is as follows: detecting significant regions in the image to be matched, encoding and transforming the detected regions to obtain image region features; for the obtained image region features, using a bottom-up attention mechanism to obtain an image global feature by performing attention weighted averaging on each local region in the image; Alternatively, performing a matching calculation based on the obtained similarities, where the similarities include the similarity matrix with implicit relationships, the similarity of the metric word segments, and the weighted similarity of the word segments.
3. The graphic-text matching method based on the fusion of attention and similarity under graph reasoning according to claim 1, wherein The feature extraction of the text includes: The words in the text to be matched are one-hot encoded through a learnable word embedding layer and mapped into a distributed representation; Using a bidirectional GRU to encode the forward and backward information to obtain enhanced word-level features; Using the average eigenvalue as a query to obtain the global representation of the entire text as a sentence-level feature.
4. The image-text matching method based on attention and similarity fusion under graph reasoning as claimed in claim 1 is characterized by: The embedding operation on the image features and the text features includes the following steps: Mapping the image features and the text features through an embedding layer to obtain a local image embedding vector and a text embedding vector; Obtaining a global image embedding vector by averaging the local image embedding vector, and obtaining a global text embedding vector by performing global weighted averaging on the text embedding vector; Expanding the global image embedding vector so that the shape of the expanded global image embedding vector is consistent with the embedding vector of the image region.
5. The graphic-text matching method based on the fusion of attention and similarity under graph reasoning according to claim 1, wherein: Using self-attention graph reasoning to capture the association, semantic relationship, and context information between the image and the text by learning the node representation and edge weights on the graph structure.
6. The graphic-text matching method based on the fusion of attention and similarity under graph reasoning according to claim 1, characterized in that: Bipolar attention simultaneously focuses on the mismatched and matched segments in the image-text pair, and measures the influence in negative attention and positive attention by adopting different attention masks.
7. A graphic-text matching system based on the fusion of attention and similarity under graph reasoning, characterized in that It includes: Feature extraction module: Configured to separately extract features from the text and picture to be matched, obtaining word-level features, sentence-level features, image region features, and image global features; Similarity calculation module: Configured to calculate the similarity between the image region features and the word-level features to obtain the local similarity, calculate the similarity between the image global features and the sentence-level features to obtain the global similarity, and perform alignment and fusion to obtain the similarity matrix; The method for calculating and aligning and fusing the local similarity and the global similarity includes the following steps: performing an embedding operation on the image features and the text features to obtain a local image embedding vector, a text embedding vector, a global image embedding vector, and a global text embedding vector; obtaining a common feature vector by performing element-wise multiplication on the local image embedding vector and the global image embedding vector; obtaining an attention weight by normalizing the weights, and obtaining an updated global image embedding vector by performing weighted summation of the attention weight and the common feature vector; constructing the obtained embedding vectors for alignment as: adopting the SCAN algorithm to construct a local-global aligned image context between the text and the image; Based on the aligned embedding vectors, perform similarity calculation and alignment fusion; based on the image region features and the text context information , calculate the local similarity between the image and the text , and the formula is: ; where is the set learnable first parameter matrix; according to the image global feature and the text global feature , calculate the global similarity between the image and the text , and the formula is: ; where is the set learnable second parameter matrix; Connect the global local similarity and the local similarity to obtain a similarity matrix , and then flatten the similarity matrix into a one-dimensional vector; ; among them, represents the matching confidence; Self-attention graph reasoning module: Configured to represent the image and text data as a graph structure based on the similarity matrix, and adopt self-attention graph reasoning to obtain a similarity matrix with implicit relationships; Bipolar attention module: configured to use soft threshold control of bipolar attention, adopt bipolar attention to focus on the mismatched and matched segments in the image-text pair, and obtain the similarity of metric word segments and the weighted similarity of word segments; bipolar attention calculation includes the following steps: calculating the semantic correlation scores between all text words and image regions; ; where, is the embedding vector of the j-th text, is the embedding vector of the i-th image region, which is also the regional feature of the image; According to the relevance score, negative attention is calculated by computing the similarity of each word segment with all image regions similarity , ; is the optimal boundary, and the negative attention is calculated as: ; and it is propagated within the text modality; according to the fact that semantically similar word segments have the same matching relationship, the matching degree of each word is propagated within the modality: ； Among them, represents the semantic relationship between the j-th word segment and the l -th word segment ; According to the correlation score, perform positive attention calculation, aggregate the matching image regions for each query word, calculate the similarity of the metric word segments and the weighted similarity of each word segment in the text; Fusion matching module: Configured to perform matching calculations based on the obtained similarities to obtain the image-text matching result; fuse the bipolar attention result and the similarity to obtain the final score. The specific fusion method is as follows: The similarity of the matching text-image pair is calculated by the formula: = + ; represents the similarity of the metric segment ; represents the weighted similarity of the word segment ; represents the similarity matrix with implicit relationships; Total similarity between the image to be recognized and the text Is the weighted sum of the similarities. To obtain the matching result of the image and text, the calculation formula is: .
8. An electronic device, characterized in that, It includes a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps in the image-text matching method based on the fusion of attention and similarity under graph reasoning according to any one of claims 1-6 are completed.
9. A computer-readable storage medium, characterized in that, Used to store computer instructions, and when the computer instructions are executed by the processor, the steps in the image-text matching method based on the fusion of attention and similarity under graph reasoning according to any one of claims 1-6 are completed.