Semantic matching method for image and text

Abstract

The invention discloses a semantic matching method for an image and a text, and belongs to the field of artificial intelligence. The invention provides an image text matching method based on credible heuristic graph learning in order to solve the ubiquitous problems of weak interpretability, poor transportability and insufficient matching result reliability existing in an existing image text matching method. The method comprises the following steps: mapping an image and a text to a common graph representation space; searching and mining a causal structure of cross-modal semantic association through a heuristic anti-fact graph; and finally, integrating evidences through a cross-modal credible alignment mechanism and quantifying the matching credibility. The method is used for mutual retrieval of images and texts, cross-modal semantic understanding and downstream task enhancement, and has the advantages of interpretable decision process, convenient module plug and play integration, robust and reliable matching result and the like.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically to a semantic matching method for images and text. Background Technology

[0002] With the explosive growth of multimedia data on the internet, achieving accurate semantic matching between images and text has become a core technology for numerous applications such as information retrieval, content recommendation, and intelligent question answering. As a core scenario for image-text retrieval, this technology aims to uncover the inherent semantic relationships between heterogeneous images and text carriers and return semantically relevant matching samples. Image-text matching is not only widely used in internet information retrieval, but its advanced technology also provides important references for downstream cross-modal tasks such as image question answering and image generation.

[0003] Existing image-text matching methods mainly fall into two paradigms: global-level methods and region-level methods. Global-level methods utilize deep neural networks to learn dense global representations of image and text samples, achieving coarse-grained alignment in the latent common space to make cross-modal semantic pairs closer. These methods are simple in structure and computationally efficient, but struggle to capture fine-grained semantic associations. Conversely, region-level methods focus on establishing fine-grained correspondences between image regions and text words. By designing complex attention mechanisms or interaction models, they explicitly learn and integrate cross-modal associations between local features. Because they can more effectively mine detailed information from samples, region-level methods typically exhibit superior matching performance and have become the mainstream research direction in this field.

[0004] Despite significant progress in matching accuracy, region-symbol fine-grained methods still face challenges and inherent limitations in practical, high-reliability applications. Most existing methods rely on deep fine-grained classifiers to extract statistical correlations from data, lacking modeling and analysis of underlying causal relationships. While this learning paradigm can capture surface associations, it struggles to address the causal mechanisms behind semantic matching, exhibits insufficient generalization ability when faced with out-of-distribution samples or noisy data, making the model susceptible to misleading spurious associations in the data, and resulting in poor interpretability of the decision-making process.

[0005] Secondly, effectively representing the complex semantic relationships among massive samples has become a significant bottleneck restricting technological development. Existing methods typically treat image regions and text words as independent feature vectors or sets, and simulate their association through attention weight matrices. This approach fails to explicitly model the inherent structured semantic relationships within and across modalities, such as the spatial and functional relationships between objects in images, and the grammatical and logical dependencies between words in text. For matching tasks that require understanding complex scenes, multiple entities, and their interactions, this flattened representation limits the model's ability to understand deep semantics.

[0006] Furthermore, existing methods generally lack the ability to quantitatively assess the reliability of matching decisions. The vast majority of existing models only output a definite matching score, without assessing the uncertainty or confidence level of the decision itself. When dealing with semantically ambiguous, information-deficient, or noisy samples, the models cannot provide a reliability metric, which may pose potential risks in semantically ambiguous or information-deficient scenarios, limiting the application of this technology in fields with high reliability requirements.

[0007] From an engineering practice perspective, current advanced matching models are mostly highly customized end-to-end architectures, with their design tightly coupled to specific tasks, datasets, and even hardware environments. This tight coupling results in poor model portability and flexibility. When integrating new technologies into existing systems or adapting to new business domains, extensive model reconstruction and parameter retuning are often required, sometimes even requiring complete retraining. This not only leads to high R&D and deployment costs but also hinders the rapid transformation and large-scale application of technological achievements. Especially against the backdrop of the rapid development of multimodal large-scale model technology, how to enhance the advanced semantic reasoning and trusted computing capabilities of existing systems in a lightweight and modular manner has become an urgent need for the industry.

[0008] Therefore, there is an urgent need in this field for a new technical solution that can enhance the model's causal understanding ability, structured semantic representation ability, and decision credibility measurement ability while maintaining the model's lightweight and plug-and-play characteristics. Summary of the Invention

[0009] To alleviate or partially alleviate the above-mentioned technical problems, the solution of the present invention is as follows:

[0010] On one hand, this invention discloses a semantic matching method for images and text, comprising the following steps:

[0011] Step S1: Obtain image samples and text samples to be matched, and extract image region features of the image samples and text word features of the text samples respectively;

[0012] Step S2: Construct a visual graph based on the image region features and a text graph based on the text word features, and map the visual graph and the text graph to a common graph representation space to obtain a common graph;

[0013] Step S3: Perform a counterfactual graph search on the public graph to generate at least one counterfactual graph, and obtain counterfactual evidence based on the counterfactual graph; wherein the counterfactual graph and the public graph have opposite semantic classification results in a preset graph classification model;

[0014] Step S4: Fuse the initial evidence obtained based on the public graph with the counterfactual evidence, perform cross-modal credibility alignment, and output the matching result of the image sample and the text sample and the corresponding credibility.

[0015] In one type of embodiment, a graph pooling step is also included:

[0016] The image region features and the text word features are respectively rearranged in descending order of their values;

[0017] The rearranged features are fed into a graph convolutional network to learn the weights of each feature element;

[0018] The rearranged features are weighted and fused based on the learned weights, and the fused optimized features are used to update or construct the common graph.

[0019] In one embodiment, performing a counterfactual graph search on the public graph includes:

[0020] The public graph is input into a graph neural network classifier to obtain a first classification result;

[0021] The edge set of the public graph is iteratively edited until the second classification result obtained by the candidate graph in the graph neural network classifier is the opposite of the first classification result.

[0022] The editing operation includes adding or deleting edges, and each editing operation selects edges based on a preset weight function.

[0023] In one embodiment, the weighting function is defined based on the consistency between the occurrence of edges in historical data and the classification labels:

[0024] ,

[0025] Wherein, for the common graph G=(E, N), E is its edge set, and N is its vertex set; e is either E or N. 2 An edge in \E; This represents the set of training graphs containing edge e and whose labels are consistent with the first classification result; This represents a set of training graphs containing edge e but whose labels are inconsistent with the first classification result.

[0026] In one embodiment, the iteratively performing an edit operation on the edge set of the common graph includes:

[0027] Before each editing operation, a random decision function is used to determine whether to add or delete an edge with equal probability.

[0028] Maintain a list of edges that have been operated on to avoid repeatedly editing the same edge;

[0029] When performing edge addition or deletion operations on the edge set of the public graph, edge addition operations are performed first.

[0030] In one embodiment, the objective of the counterfactual graph search is to find the counterfactual graph with the minimum edit distance to the common graph; wherein the edit distance is measured by the symmetric difference of the edge sets of the two graphs.

[0031] In one embodiment, in step S4, the initial evidence is obtained in the following manner:

[0032] The similarity score s between the image region features and the text word features is mapped to a non-negative evidence value using the evidence transformation function g(.). ;

[0033] ;

[0034] Where P(.) is a nonlinear activation function, including the ReLU function, exponential function or Softplus function, and τ is a temperature parameter with a value range of (0,1).

[0035] In one embodiment, the cross-modal trusted alignment in step S4 specifically includes:

[0036] The initial evidence is added to the counterfactual evidence to obtain the total evidence;

[0037] Based on the total evidence, the consistency loss L in the image-to-text direction is calculated respectively. i2t And text-to-image orientation consistency loss L t2i Among them, L i2t Calculation of the difference between the evidence output by the image-to-text retrieval model and the evidence output by the text-to-image retrieval model on the same task; L t2i The calculation is similar;

[0038] The consistency loss is used to constrain the uncertainty estimates of bidirectional forecasts to tend towards consistency.

[0039] The consistency loss from the image to the text direction and the consistency loss from the text to the image direction are added together to obtain the total loss, and the model parameters are updated by optimizing the total loss.

[0040] On the other hand, the present invention discloses an electronic device, comprising:

[0041] One or more processors;

[0042] Memory, used to store one or more programs;

[0043] When the one or more programs are executed by the one or more processors, the one or more processors implement the semantic matching method for images and text as described in any of the preceding items.

[0044] On the other hand, the present invention discloses a computer-readable storage medium having a computer program stored thereon:

[0045] When executed by a processor, this computer program implements the semantic matching method for images and text as described in the previous one.

[0046] The technical solution of this invention has one or more of the following beneficial technical effects:

[0047] (1) Significantly improves the interpretability and reliability of model decision-making. By introducing a causal reasoning mechanism, the model can go beyond traditional correlation matching and gain a deeper understanding of the inherent logic and decision boundaries of the association between images and text. This makes the matching results not only accurate, but also transparent and traceable, enhancing its credibility in practical applications.

[0048] (2) It achieves excellent versatility and convenient integration and deployment. This solution adopts a "plug-and-play" modular design, which can flexibly and efficiently integrate with various existing image-text matching models without modifying the core structure, significantly improving their performance. This greatly reduces the cost of replicating and deploying advanced technologies, providing a cost-effective solution for industrial applications.

[0049] (3) Effectively enhances the robustness and anti-interference ability of cross-modal matching. By constructing a unified credibility assessment and alignment mechanism, the model can autonomously quantify the uncertainty in the matching process and impose consistency constraints on predictions in different directions, thereby outputting more stable and reliable results when facing noisy data or complex scenarios, and improving the overall practical value of the system.

[0050] Furthermore, other beneficial effects of the present invention will be mentioned in the specific embodiments. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating one embodiment of the present invention;

[0052] Figure 2 This is an overall system architecture diagram of one embodiment of the present invention;

[0053] Figure 3 This is a diagram comparing counterfactual graph search with traditional search methods;

[0054] Figure 4 This is a diagram comparing graph pooling with traditional pooling methods;

[0055] Figure 5 This is a graph showing the effect of the temperature parameter τ on the model's performance.

[0056] Figure 6 This is a cross-modal retrieval case of one embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order.

[0059] This invention proposes a semantic matching method for images and text. By unifying the representation of image regions and text symbols as structured graphs and introducing a counterfactual graph search mechanism for causal reasoning, it reveals the key semantic associations in the matching decision. Simultaneously, it combines a cross-modal trusted alignment framework to quantify matching uncertainty and supplements it with adaptive graph pooling to optimize feature representation. Finally, while improving matching accuracy, it outputs a trusted result that combines matching score and confidence, significantly enhancing the model's generalization, interpretability, and practical deployment flexibility. Figure 1 This is a flowchart illustrating one embodiment of the present invention. The diagram clearly demonstrates the complete processing flow of a semantic matching method for images and text provided by the present invention.

[0060] This invention specifically provides an image-text matching enhancement method based on trusted heuristic graph learning. Figure 2 This is a diagram illustrating the overall system architecture of one embodiment of the present invention. The overall system architecture includes an image sampling network, a text sampling network, a common graph representation module, a heuristic graph learning module, and a cross-modal reliable alignment module. As shown in the diagram, the system receives pairs of images and text as input. First, it extracts structured features from the input images and text using parallel image and text sampling networks.

[0061] First, given a dataset containing n image-text pairs:

[0062] ,

[0063] in, For image samples, This is a text sample. For each image (img), the parameters are respectively... The image sampling network and parameters are as follows Feature extraction is performed using a text sampling network. (The last part, "i," appears to be incomplete and lacks context. It x Let t represent the feature vector of the x-th region in the image img, where t y This represents the feature vector of the y-th word in the text txt. The image img is encoded as W region features:

[0064] ,

[0065] Similarly, for the text txt, it is encoded into Z token features:

[0066] .

[0067] Next, the extracted modality-specific features I and T are mapped to a common feature space to construct visual graphs G for images and text, respectively. i and text image G t This forms a unified graph structure representation. Then, its cross-modal similarity is calculated. The similarity function is defined as:

[0068]

[0069] Where h(.) is the similarity function (such as cosine similarity, vector similarity, and mixed similarity), and different internal sub-modules can adopt different measurement strategies. This is its parameter set. This function is used to measure the degree of semantic association between an image and text.

[0070] In the training process of the method described in this invention, each batch of training data is randomly sampled from the complete dataset D, denoted as... This batch processing mechanism helps the model learn efficiently and enhances its generalization ability. During model training, to learn effective matching representations, this invention references a general bidirectional ranking loss as the basic objective in the initial stage. For a matching pair of image and text (I, T), the hardest negative sample within the batch is defined as... and The two-way ranking loss is defined as:

[0071] ,

[0072] in, For the edge moment parameters, [...] ＋ This represents a function that takes positive values.

[0073] Furthermore, to introduce credible learning and uncertainty quantification, this invention proposes converting similarity scores into nonnegative evidence. Given a similarity score s, it is mapped to a nonnegative evidence value using the evidence transformation function g(.). :

[0074] ,

[0075] P(.) can be a nonlinear activation function such as ReLU, exp(.), or Softplus. The temperature parameter is used to adjust the intensity of evidence scaling. For a series of similarity scores s1, s2, ..., s... m The corresponding evidence vector can be obtained. , where m is the number of heuristic edges considered.

[0076] Based on public graph representations, the system performs heuristic graph learning, specifically manifested as a counterfactual graph search process.

[0077] Figure 3 This diagram illustrates a comparison between counterfactual graph search and traditional search methods. As shown, it highlights the fundamental differences between the counterfactual graph search proposed in this invention and existing traditional and K-way search methods. Traditional methods typically query along a single or fixed path; while the K-way search method considers multiple paths and Dirichlet distributions, it still requires multiple traversals and has low efficiency. In contrast, the counterfactual graph search of this invention can complete the search process in one step and effectively optimize the partitioning of nodes in the graph.

[0078] First, construct a simple graph with no weights from all nodes. For a set of vertices N, N 2 Represents all possible The set of pairs, i.e. In the actual constructed graph G=(E, N), the edge set E is N. 2 subsets of (E⊆N) 2 This invention defines a graph matching prediction function (or black-box predictor) f: G(N) → {0,1}, whose initial parameters are random, and can be used to determine whether a graph belongs to a semantically related class.

[0079] The objective of counterfactual graph search is to find an edge set E for the original graph G = (E, N). * This makes the new graph G * =(E * The classification result of (N) in the predictor is the opposite of the original image, and the edit distance of the edge set is minimized. This problem can be formalized as:

[0080] ,

[0081] in, It is a symmetric difference, used to measure the degree of change in a boundary set. And... , This represents the symmetric difference between two edge sets. Since the search space size grows exponentially with the number of vertices n... Directly solving this problem is too costly. Therefore, this invention designs a heuristic graph local search algorithm (corresponding to Table 1). This process begins with an initial graph G and a black-box predictor f. Its core is to iteratively edit the edge set of graph G (adding or deleting edges) until a counterfactual graph G is generated. c The predictor f classifies the data as the opposite of the original graph.

[0082] In summary, the common graph obtained by extracting features and constructing a graph to represent cross-modal semantic associations is input into a pre-trained or randomly initialized graph neural network classifier (hereinafter referred to as a black-box graph predictor). After obtaining the first classification result, the edge set of the common graph is iteratively edited until the second classification result obtained by the candidate graph after editing in the black-box graph predictor is the opposite of the first classification result.

[0083] To avoid invalid search traversals during this process, the system maintains a list L of operated edges to ensure that the same edge is not manipulated repeatedly. The decision for an edit operation is guided by a stochastic process: before each operation, a random, unbiased coin toss function `coin(.)` determines whether the operation involves adding or deleting an edge. To more efficiently approach the target, the strategy of this invention tends to prioritize edge-adding operations. The detailed steps of this process are listed in Table 1, constituting an operational implementation of the counterfactual graph search.

[0084] Table 1: Counterfactual Map Search

[0085]

[0086] To further optimize the search direction and improve the efficiency and quality of generating counterfactual graphs, this invention introduces a weighted guidance mechanism based on data statistics. On the multimedia database D, in each iteration, the edge distribution can be updated to distinguish between relevant and irrelevant instances. Biased random selection (coin(.)) is used to add or remove edges to better differentiate the edge distribution. Specifically, a weighting function is defined. The function is based on each possible edge e∈N 2 The occurrence of an edge in historical data is used to assign a weight value, which guides the selection priority during editing. The weight calculation method for edge e is as described above:

[0087] ,

[0088] in, , represents the set of training graphs containing edge e and whose labels are consistent with the original graph f(e) (the first classification result).

[0089] Let f(e) represent the set of training graphs containing edge e but whose labels are inconsistent with the original graph f(e) (the first classification result). E is the set of edges of the common graph, and N is the set of vertices of the common graph.

[0090] This strategy involves removing edges with strong semantic relevance from the edge set E, which makes it easier to generate counterfactual graphs with smaller edit distances, or adding potentially irrelevant edges E. In implementation, certain settings are used to prevent zero-probability edges while making negatively weighted edges less likely to be selected. Specifically, when deciding to add or remove an edge, the system adjusts the probability of different edges being selected based on their weights w(e), ensuring that edges more likely to guide the graph structure across the classification boundary are manipulated more frequently. This mechanism ensures that the search process not only relies on random exploration but also leverages learned knowledge of data distribution, thus finding the counterfactual graph with the minimum edit distance more intelligently and efficiently.

[0091] To further enhance feature representation capabilities, this invention proposes a graph pooling mechanism. Figure 4 This is a diagram comparing graph pooling with traditional pooling methods. As shown in the figure, unlike traditional max pooling or average pooling, graph pooling first rearranges the image region features I and text word features T in descending order of value to form I0. * and T * The re-acquired features (e.g., A={4,3,5,6,4}, B={3,2,4,4,3}) are then incorporated into a graph convolutional network to aggregate all non-maximum feature elements added during pooling, thereby preserving more local structural information and enhancing cross-modal semantic alignment capabilities. This process can be represented as:

[0092] ,

[0093] ,

[0094] The k-th element represents the k-th max-pooling result. This graph pooling mechanism constructs an improved graph convolutional network framework, which uses intra-node self-attention instead of fully connected node feature propagation operations to represent the obtained features as nodes in the pooling network and constructs a fully connected graph of cross-modal features. In the constructed graph pooling network, both image and text merging layers are set to 1. Each weight pass calculation takes node features as input, performing convolution, dot product, and normalization sequentially to obtain a specific weight for each element. Through this design, the weights of all reordered local features can be calculated, capturing more complex internal relationships and enhancing feature representation capabilities. In other words, the model can adaptively capture complex relationships between features, achieving finer-grained semantic alignment.

[0095] Finally, in the cross-modal trusted alignment module, the system integrates evidence obtained from traditional search and heuristic graph search. That is, the total evidence consists of non-negative evidence obtained from traditional search. Non-negative evidence obtained from graph search Integrated:

[0096] ,

[0097] in It is the non-negative evidence generated from the first search, and It is non-negative evidence obtained from heuristic graph search.

[0098] To reduce inconsistencies in model predictions during bidirectional learning, this invention introduces consistency regularization loss. Specifically, this invention designs two independent learning models, each focusing on a specific learning direction for different retrieval tasks: one for image-to-text and the other for text-to-image. Since the differences between these directions will inevitably lead to inconsistent uncertainty estimates from different models in the same direction—for example, given a query, one model might produce a low-uncertainty prediction, while the other might have a higher uncertainty—this invention introduces consistency regularization to force the two query models to produce consistent predictions of subjective opinions. Specifically, it utilizes the L1 norm to measure and minimize the differences in subjective opinions predicted by the two task-specific query models.

[0099] Taking the image-to-text direction as an example, the consistency loss is defined as:

[0100]

[0101] in, and The optimized formula yields:

[0102] ,

[0103] in, and It is the uncertainty estimate of the image-to-text direction for image and text query models.

[0104] The text-to-image direction loss is similar. The total bidirectional loss is:

[0105]

[0106] Finally, the model is optimized by minimizing the total loss. The optimization process is shown in Table 2: data is randomly sampled in each training batch, features are extracted, forward propagation is performed to calculate the loss, and the network parameters are updated through backpropagation. and The optimizer can employ stochastic gradient descent algorithms such as Adam or SGD.

[0107] Table 2: Model Optimization Process

[0108]

[0109] Through the collaborative work of the above modules, this invention achieves plug-and-play, causal enhancement, and graph representation-guided image-text matching, significantly improving matching performance and model portability.

[0110] To verify the effectiveness of the proposed method, extensive experiments were conducted and it was compared with several state-of-the-art image-text matching methods on two widely used benchmark datasets, namely MSCOCO and Flickr30K datasets.

[0111] Quantitative experiments were first conducted. Through trusted heuristic graph learning, this method is applicable to various retrieval scenarios. Tables 3 and 4 show a quantitative comparison of recall between the method of this invention and state-of-the-art image-text matching methods. Specifically, eight image-text matching methods were compared. Like the method of this invention, three of the control methods are also plug-and-play (AdaCL, FDF, and NDGL). To ensure fairness, the performance improvements of these four methods based on the CHAN baseline were compared. All experimental results are based on two sampling backbones: BUTD+Bi-GRU and BUTD+BERT.

[0112] Table 3: Quantitative experiments on the MSCOCO dataset

[0113]

[0114] Table 4: Quantitative experiments on the Flickr30K dataset

[0115]

[0116] Table 3 shows the retrieval results of the model of this invention on the MSCOCO dataset, including the MSCOCO 1K test and the MSCOCO 5K test. The model of this invention significantly outperforms state-of-the-art methods, achieving optimal values ​​on different retrieval tasks. Table 4 compares the results of existing methods on the bidirectional retrieval task on the Flickr30K dataset. Experimental results show that the method of this invention outperforms most other methods. Based on the overall performance of the two datasets, the design of this invention is innovative, using graph representation and heuristic search to explore the intrinsic invariant principle for image-text matching tasks. Furthermore, this invention (THGL) exhibits good transferability and can effectively improve the matching performance of the original baseline.

[0117] Based on the experimental results, further analysis was conducted. First, in terms of overall retrieval performance, the method of this invention is more stable and exhibits superior overall performance. Second, compared to the initial baseline CHAN, the introduction of the module (THGL) of this invention significantly improved the initial performance, demonstrating that the design of this invention possesses sufficient innovation and portability. Third, compared to the other three plug-and-play methods based on CHAN (AdaCL, FDF, and NDGL), the performance of this invention is more outstanding and has advantages in similar designs.

[0118] Then, a comparison with different baselines for integrating the invention's modules (THGL) is presented. To verify the transferability of the invention, training and testing results integrated with four baseline models are discussed, namely... ESA, HERM, and CHAN are shown in Tables 5 and 6. Table 5 shows the performance improvement of different baselines on the MSCOCO dataset after integrating the proposed module (THGL), including the MSCOCO 1K and MSCOCO 5K tests. Table 6 shows the performance improvement of different baselines on the Flickr30K dataset after integrating the proposed module (THGL). Similar to the aforementioned performance comparisons, the settings of the BUTD+Bi-GRU and BUTD+BERT sampling backbones are also discussed. The numbers in parentheses indicate the performance improvement of the original methods after integrating the framework proposed in this invention. The experimental results show that the proposed solution can provide semantic alignment from more perspectives, further improving the accuracy of image-text matching.

[0119] Table 5: Performance improvements after integrating THGL with different baselines on the MSCOCO dataset

[0120]

[0121] Table 6: Performance improvements after integrating THGL with different baselines on the Flickr30K dataset

[0122]

[0123] Next, we perform hyperparameter analysis, specifically τ in the aforementioned formula. The experiment was conducted on Flickr30K to demonstrate the overall accuracy Rsum of the bidirectional matching task. The experimental results are as follows: Figure 5 As shown, the ideal range for τ is between 0.03 and 0.07, since similarity is typically between 0 and 1, and appropriate scaling facilitates computation. Small values ​​of τ may make cross-modal models difficult to optimize, leading to poor performance; too large a value (close to 1) will limit previous similarity scaling, causing τ to fail. Therefore, a suitable τ can bring the proposed framework to its optimal value.

[0124] To visualize the improvements of this invention, instance retrieval analysis was performed using the original CHAN model and the CHAN integrated with the modules of this invention (i.e., CHAN+THGL). Figure 6 This is a cross-modal retrieval case of one embodiment of the present invention. As shown in the figure, several examples of text-to-image and image-to-text retrieval results obtained using two strategies on the MSCOCO dataset are displayed. First, a matching image is searched using the input term, such as... Figure 6 As shown in part (a). The trusted graph learning method designed in this invention can mine complex regions, thereby improving the performance of the ensemble model (the original model's first row, fourth result did not contain "airplane"). Secondly, this invention uses the input image to find the corresponding text, such as Figure 6 As shown in section (b), the proposed image semantic matching module can effectively perform cross-modal semantic alignment. Based on the results of the two matching tasks, the retrieval performance of the improved model is enhanced compared to the original model after integrating this invention. It can be seen that the design of this invention is beneficial to the overall model.

[0125] Experimental results show that the present invention outperforms existing mainstream methods on multiple benchmark datasets, verifying its effectiveness and advancement.

[0126] In summary, this invention constructs an efficient, reliable, and highly interpretable image-text matching enhancement system by organically integrating a series of innovative technical modules, including structured public graph representation, heuristic counterfactual graph search, cross-modal trusted alignment, and adaptive graph pooling. This method adopts a plug-and-play design philosophy, enabling flexible modular integration into various existing image-text matching models. Without significantly increasing computational complexity, it systematically improves the semantic understanding depth, anti-interference ability, decision transparency, and final application reliability in complex scenarios.

[0127] To better illustrate the present invention, numerous specific details have been provided in the detailed embodiments described above. Those skilled in the art should understand that the present invention can be practiced even without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of the present invention.

[0128] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A semantic matching method for images and text, characterized in that, Includes the following steps: Step S1: Obtain image samples and text samples to be matched, and extract image region features of the image samples and text word features of the text samples respectively; Step S2: Construct a visual graph based on the image region features and a text graph based on the text word features, and map the visual graph and the text graph to a common graph representation space to obtain a common graph; Step S3: Perform a counterfactual graph search on the public graph to generate at least one counterfactual graph, and obtain counterfactual evidence based on the counterfactual graph; wherein the counterfactual graph and the public graph have opposite semantic classification results in a preset graph classification model; Step S4: Fuse the initial evidence obtained based on the public graph with the counterfactual evidence, perform cross-modal credibility alignment, and output the matching result of the image sample and the text sample and the corresponding credibility.

2. The semantic matching method for images and text according to claim 1, characterized in that, It also includes the graph pooling step: The image region features and the text word features are respectively rearranged in descending order of their values; The rearranged features are fed into a graph convolutional network to learn the weights of each feature element; The rearranged features are weighted and fused based on the learned weights, and the fused optimized features are used to update or construct the common graph.

3. The semantic matching method for images and text according to claim 1, characterized in that, The reverse fact graph search performed on the public graph includes: The public graph is input into a graph neural network classifier to obtain a first classification result; The edge set of the public graph is iteratively edited until the second classification result obtained by the candidate graph in the graph neural network classifier is the opposite of the first classification result. The editing operation includes adding or deleting edges, and each editing operation selects edges based on a preset weight function.

4. The semantic matching method for images and text according to claim 3, characterized in that, The weighting function is defined based on the consistency between the occurrence of edges in historical data and the classification labels: ， Wherein, for the common graph G=(E, N), E is its edge set, and N is its vertex set; e is either E or N. 2 An edge in \E; This represents the set of training graphs containing edge e and whose labels are consistent with the first classification result; This represents a set of training graphs containing edge e but whose labels are inconsistent with the first classification result.

5. The semantic matching method for images and text according to claim 3, characterized in that, The iterative editing operation on the edge set of the common graph includes: Before each editing operation, a random decision function is used to determine whether to add or delete an edge with equal probability. Maintain a list of edges that have been operated on to avoid repeatedly editing the same edge; When performing edge addition or deletion operations on the edge set of the public graph, edge addition operations are performed first.

6. The semantic matching method for images and text according to claim 4, characterized in that: The objective of the counterfactual graph search is to find the counterfactual graph with the minimum edit distance to the common graph; wherein the edit distance is measured by the symmetric difference of the edge sets of the two graphs.

7. The semantic matching method for images and text according to claim 1, characterized in that, In step S4, the initial evidence is obtained in the following manner: The similarity score s between the image region features and the text word features is mapped to a non-negative evidence value using the evidence transformation function g(.). ; ； Where P(.) is a nonlinear activation function, including the ReLU function, exponential function or Softplus function, and τ is a temperature parameter with a value range of (0,1).

8. The semantic matching method for images and text according to claim 7, characterized in that, The cross-modal reliable alignment in step S4 specifically includes: The initial evidence is added to the counterfactual evidence to obtain the total evidence; Based on the total evidence, the consistency loss L in the image-to-text direction is calculated respectively. i2t And text-to-image orientation consistency loss L t2i ; among which, L i2t Calculation of the difference between the evidence output by the image-to-text retrieval model and the evidence output by the text-to-image retrieval model on the same task; L t2i The calculation is similar; The consistency loss is used to constrain the uncertainty estimates of bidirectional forecasts to tend towards consistency. The consistency loss from the image to the text direction and the consistency loss from the text to the image direction are added together to obtain the total loss, and the model parameters are updated by optimizing the total loss.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the semantic matching method for images and text as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the semantic matching method for images and text as described in any one of claims 1 to 8.

Images