A noise-robust multi-modal image-text matching method based on multi-evidence fusion

Abstract

The application provides a noise-robust multi-modal image-text matching method based on multi-evidence fusion, and belongs to the technical field of data processing. Specifically, cross-modal similarity evidence and same-modal topological structure evidence of an image-text sample pair are extracted, two types of evidence are fused by using an improved Dempster-Shafer combination rule, and fused evidence and corresponding uncertainty measurement are obtained; based on the uncertainty measurement, a training set is divided into clean samples and noise samples, the clean samples are subjected to parameter updating, and a topological preservation loss is used to optimize global topological structure of an embedding space, the noise samples are subjected to adaptive re-matching based on a partial optimal transport model to mine potential semantic association and update model parameters; after training is completed, cross-modal similarity of a target image-text sample pair is calculated, and a matching result is output. Through the scheme, the precision and robustness of image-text matching in a noise scene are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a noise-robust multimodal image-text matching method based on multi-evidence fusion. Background Technology

[0002] Currently, image-text matching is a fundamental technology for tasks such as cross-modal retrieval and visual question answering. It aims to learn a shared embedding space to measure the semantic consistency between images and text. However, a large amount of weakly aligned image-text data on the Internet naturally contains semantic mismatch noise. When the noise ratio exceeds 20%, the retrieval accuracy of mainstream models can drop by 10 to 20 percentage points. To address this issue, existing technologies mainly include sample partitioning methods based on probabilistic mixture models, partition-free methods based on complementary learning, and uncertainty modeling methods based on evidence-based deep learning, supplemented by optimization techniques such as soft labeling and optimal transmission re-matching. However, these solutions generally suffer from the following shortcomings: probabilistic mixture models rely solely on single-dimensional loss or similarity information for partitioning, failing to utilize the topological features of the embedding space, and the dual-network collaborative mechanism significantly increases training costs; complementary learning relies solely on negative sample supervision, which is prone to underfitting due to weak signals, and its uniform distribution assumption is not applicable to real-world dynamic noise scenarios; existing evidence-based deep learning uses only a single cross-modal similarity as the source of evidence, lacking a multi-dimensional fusion mechanism, and fails to address the problem of rare semantic loss caused by evidence underestimation; the core parameters of re-matching methods need to be manually set and lack adaptive capabilities, and most solutions only optimize point-to-point similarity while ignoring global topological constraints, making it difficult to balance high accuracy, strong robustness, and low deployment costs.

[0003] It is evident that there is an urgent need for a noise-robust multimodal image-text matching method based on multi-evidence fusion that offers both high matching accuracy and robustness. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a noise-robust multimodal image-text matching method based on multi-evidence fusion, which at least partially solves the problems existing in the prior art.

[0005] This invention provides a noise-robust multimodal image-text matching method based on multi-evidence fusion, comprising: Step 1: Obtain multimodal image-text sample pairs, wherein the multimodal image-text sample pairs include images and corresponding text descriptions; Step 2: Input the multimodal image-text sample pairs into the basic matching model to extract the first modal features of the image and the second modal features of the text description; Step 3: Based on the first modal features and the second modal features, extract cross-modal similarity evidence and same-modal topological structure evidence for each multimodal image-text sample; Step 4: Input cross-modal similarity evidence and same-modal topological structure evidence into the evidence fusion module, and fuse them according to the improved Dempster-Shafer combination rule to obtain fused evidence and corresponding uncertainty measure; Step 5: After the warm-up phase of model training is completed, the sample pairs in the training set are divided based on the uncertainty metric to obtain a clean sample subset and a noisy sample subset. Step 6: In the first stage of model training, the parameters of the basic matching model are updated using a clean sample subset, and the global topology of the embedding space of the image and text descriptions is optimized collaboratively using topology preservation loss. Step 7: In the second stage of model training, adaptive re-matching is performed on the subset of noisy samples based on the partially optimal transmission model to mine the potential semantic associations in the subset of noisy samples, and the parameters of the basic matching model are updated using the re-matching results. Step 8: After the model training is completed, obtain the target image-text sample pair, input the target image-text sample pair into the updated base matching model, calculate the cross-modal similarity between the image and the text description in the target image-text sample pair, and output the image-text matching result of the target image-text sample pair based on the cross-modal similarity.

[0006] According to a specific implementation of an embodiment of the present invention, step 3 specifically includes: Calculate the bidirectional cross-modal matching similarity between the image and text description in the multimodal image-text sample pair to obtain the cross-modal similarity matrix; Using the identified clean samples as structural base points, we calculate the neighborhood distribution distance of a single image in the image embedding space and the neighborhood distribution distance of a single text description in the text embedding space, and construct the image intramodal similarity matrix and the text intramodal similarity matrix. Calculate topological similarity based on the intra-modal similarity matrix of images and the intra-modal similarity matrix of text; The cross-modal similarity and topological similarity in the cross-modal similarity matrix are transformed independently into cross-modal similarity evidence and same-modal topological evidence through nonlinear transformation.

[0007] According to a specific implementation of an embodiment of the present invention, the step of calculating topological similarity based on the image intra-modal similarity matrix and the text intra-modal similarity matrix includes: Construct image topology vectors based on image intra-modal similarity matrices and text intra-modal similarity matrices. and text topology vector ; Calculate the cosine similarity between the image topological structure vector and the text topological structure vector, and use this as the topological similarity. .

[0008] According to a specific implementation of an embodiment of the present invention, step 4 specifically includes: First subjective opinion corresponding to cross-modal similarity evidence Second subjective opinion corresponding to the same modal topological structure evidence An improved Dempster-Shafer combination rule with an adaptive correction mechanism incorporating conflict metrics is used for fusion, and the trust quality of the fused result is... and uncertainty measurement for: ; ; in, , All for A dimensional vector of subjective opinions. As a conflict metric, it is calculated using the conflict trust quality from two subjective opinion vectors. This represents the k-th dimension of trust quality in the first subjective opinion vector. This represents the k-th dimension of trust quality in the second subjective opinion vector. This represents the uncertainty in the first subjective opinion vector. This represents the uncertainty in the second subjective opinion vector; Based on the quality of trust after integration and the uncertainty after fusion Restore the fused evidence vector; The fused evidence vector is incremented by 1 to obtain the shape parameter of the Dirichlet distribution. The Dirichlet distribution is then constructed based on the shape parameter to model the sample matching probability. Calculate the trust quality based on the recovered and fused evidence vector. and overall uncertainty : ; ; in, This is the evidence vector obtained after fusion and reconstruction. For the total amount of evidence, Let be the dimension of the evidence vector.

[0009] According to a specific implementation of an embodiment of the present invention, the first stage of training the basic matching model employs a robust loss function based on the Fisher information matrix, the robust loss function comprising: Fisher information matrix weighted mean square error term This is used to weight the traditional Bayesian risk loss, enhancing the model's attention to low-evidence sample pairs; Fisher Information Matrix Regularization Term , used to constrain the magnitude of the model's evidence output; KL divergence regularization term This is used to constrain the predicted distribution to approximate the prior distribution; The expression for the robust loss function is: ; in, , These are the weighting coefficients.

[0010] According to a specific implementation of the present invention, the weighted mean square error term of the Fisher information matrix... The expression is: ; in, It is a three-gamma function. The first of the Dirichlet distributions The parameters are calculated from the evidence vector. The sum of the parameters of the Dirichlet distribution; The Fisher information matrix regularization term The expression is: .

[0011] According to a specific implementation of an embodiment of the present invention, the topology preservation loss includes cross-consistency loss and metric consistency loss: The cross-consistency loss The expression for constraining the bidirectional cross-modal similarity between image-to-text and text-to-image is: ; The measurement of consistency loss The expression used to constrain the consistency of intramodal similarity and cross-modal similarity is: ; in, For the clean sample subset in the current batch, They are the same margin hyperparameter.

[0012] According to a specific implementation of an embodiment of the present invention, the step of adaptively re-matching a subset of noise samples based on a partially optimal transmission model includes: Based on clean and noisy sample subsets, a cross-modal similarity matrix is ​​constructed, and a cost matrix is ​​constructed based on the cross-modal similarity matrix: ; in Cross-modal similarity between sample pairs; Constructing a partially optimal transport problem with regularization constraints: ; in, For the transmission cost matrix, Denotes the Frobenius inner product of matrices. For entropy regularization, The balancing parameter for the entropy regularization term. The marginal regularization equilibrium parameter, and Let n and m be unit vectors, respectively. and Let be m-dimensional and n-dimensional uniform marginal distribution vectors, respectively, and T be the transfer matrix to be solved. The generalized Sinkhorn algorithm is used to solve a partial optimal transmission problem, resulting in a transmission plan T. The rematch probability is calculated based on the transmission plan T, and the parameters of the basic matching model are updated using the symmetric KL divergence as the rematch loss.

[0013] According to a specific implementation of the present invention, in the first stage of training the basic matching model, an asymptotic triplet loss is used to optimize the clean sample subset, and the number w of extremely difficult negative samples in the asymptotic triplet loss adaptively decreases with the training rounds: ; in, For batch size, The attenuation rate, This represents the minimum number of difficult cases.

[0014] According to a specific implementation of an embodiment of the present invention, in the warm-up stage, the basic matching model is trained using all training samples without dividing or removing noise sample pairs, and the number of training cycles in the warm-up stage is adaptively determined according to a preset noise ratio.

[0015] The noise-robust multimodal image-text matching scheme based on multi-evidence fusion in this embodiment of the invention includes: Step 1, acquiring multimodal image-text sample pairs, wherein the multimodal image-text sample pairs include images and corresponding text descriptions; Step 2, inputting the multimodal image-text sample pairs into a basic matching model, and extracting the first modal features of the images and the second modal features of the text descriptions; Step 3, based on the first modal features and the second modal features, extracting cross-modal similarity evidence and same-modal topological structure evidence for each multimodal image-text sample; Step 4, inputting the cross-modal similarity evidence and same-modal topological structure evidence into an evidence fusion module, and fusing them according to an improved Dempster-Shafer combination rule to obtain fused evidence and corresponding uncertainty metrics; Step 5, after the model training warm-up phase, adjusting the training data based on the uncertainty metrics. Step 6: In the first stage of model training, the parameters of the basic matching model are updated using the clean sample subset, and topology preservation loss is used to jointly optimize the global topology of the embedding space of the image and text description. Step 7: In the second stage of model training, adaptive re-matching is performed on the noisy sample subset based on the partially optimal transfer model to mine the potential semantic associations in the noisy sample subset, and the parameters of the basic matching model are updated using the re-matching results. Step 8: After the model training is completed, the target image-text sample pair is obtained, the target image-text sample pair is input into the updated basic matching model, the cross-modal similarity between the image and text description in the target image-text sample pair is calculated, and the image-text matching result of the target image-text sample pair is output according to the cross-modal similarity.

[0016] The beneficial effects of the embodiments of the present invention are as follows: 1. Significantly improved sample segmentation accuracy Existing deep learning methods for evidence rely solely on cross-modal similarity as the source of evidence, failing to utilize the topological features of the embedding space, resulting in insufficient sample segmentation accuracy. This solution designs a dual-dimensional multi-source evidence extractor that simultaneously extracts cross-modal similarity evidence and intramodal topological evidence. An improved Dempster-Shafer combination rule is employed to achieve multi-evidence fusion, overcoming the incompleteness of single-dimensional evidence and achieving accurate segmentation between clean and noisy samples. Simultaneously, it outputs quantifiable and traceable sample matching uncertainty. This method supports stable training in high-noise scenarios, effectively overcoming the convergence failure of some existing techniques in high-noise environments.

[0017] 2. The accuracy of image-text matching and the robustness of the model are significantly improved. Existing methods only focus on optimizing point-to-point similarity between images and text, without co-constraining the global topology of the embedding space, resulting in insufficient feature consistency for samples with the same semantic meaning. This approach uses cross-consistency loss and metric consistency loss to co-optimize the global topology of the image and text embedding spaces, strengthening the feature consistency of samples with the same semantic meaning across different modal embedding spaces. It achieves a significant improvement in the overall bidirectional image-text matching performance under all gradient noise levels, effectively suppresses performance degradation in high-noise scenarios, and significantly enhances the model's noise robustness.

[0018] 3. Training and deployment costs have been significantly reduced. To address the issue that existing probabilistic mixture model-based methods commonly employ dual-network or even multi-network collaborative teaching mechanisms, leading to a significant increase in training time and GPU memory usage, this solution adopts a single-network end-to-end architecture. Training can be completed with only one forward propagation and one backward propagation, eliminating the need for multi-network collaborative mechanisms. This greatly simplifies the model architecture and training process, significantly reduces training GPU memory usage, and is more suitable for industrial deployment in large-scale image and text data scenarios.

[0019] 4. The semantic utilization rate of noisy samples is significantly improved. Existing noise rematching methods require manual setting of core parameters, lack adaptability, and cannot fully exploit the latent semantic associations of noise samples, easily leading to the loss of rare semantic features. This solution employs an imbalanced optimal transmission model to achieve adaptive semantic rematching of noise samples. It eliminates the need for manual setting of core parameters such as transmission quality, can adaptively adapt to different noise distribution scenarios, fully exploits the latent semantics and rare attributes in noise samples, and further improves the model's generalization ability and data utilization. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a noise-robust multimodal image-text matching method based on multi-evidence fusion provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of multi-evidence fusion provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of some optimal transmission rematch strategies provided in embodiments of the present invention; Figure 4 This is a schematic diagram illustrating the classification based on uncertainty and trust quality provided for an embodiment of the present invention. Detailed Implementation

[0022] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this invention, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0025] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0026] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0027] The core objective of image-text matching is to learn a cross-modal similarity function within a shared embedding space, measuring semantic consistency between visual and linguistic modalities. It is a core upstream technology for tasks such as cross-modal retrieval, visual question answering, and image caption generation. The current internet landscape contains billions of weakly aligned image-text pairs, providing rich supervisory signals for cross-modal model training. However, this data naturally contains a certain proportion of semantic mismatch noise, i.e., noise correspondences where images and text are semantically unrelated or weakly correlated. This problem leads to overfitting of incorrect supervisory signals during model training, resulting in a significant decrease in image-text matching accuracy. When the mismatch rate in the training set exceeds 20%, the retrieval accuracy of mainstream image-text matching models drops sharply by 10 to 20 percentage points, simultaneously creating a cascading negative impact on downstream tasks. The core technical problems this invention aims to solve are: how to accurately identify noisy samples, improve the model's training robustness under noisy data, and fully exploit the semantic value of weakly aligned image-text pairs.

[0028] To address the noise correspondence problem in image-text matching, existing technologies generally adopt a "partition-learning" core framework, decoupling the problem into two progressive subtasks: partitioning clean and noisy samples without ground truth, and robustly training the model based on the partitioning results. Current mainstream solutions can be categorized as follows: Sample partitioning methods based on probabilistic mixture models. These methods leverage the memory effect of deep neural networks—that is, the higher loss to noisy samples in the early stages of training—by using probabilistic mixture models such as Gaussian mixture models and Beta mixture models to model the training loss or cross-modal similarity distribution of samples, thereby partitioning the training set into clean and noisy subsets. To avoid the accumulation and amplification of partitioning errors during training, most of these methods are equipped with a dual-network collaborative teaching mechanism. Some schemes further improve the accuracy of sample partitioning by introducing regularization terms, meta-learning to correct similarity scores, and improving the probabilistic mixture model.

[0029] Complementary learning methods that do not require explicit sample partitioning. These methods rely on the statistical characteristic that negative sample labeling error rates are lower in image-text matching tasks. That is, within a single batch, for an image sample, most text is a true negative sample, and vice versa. The model training objective is transformed into minimizing the expected probability of negative example matching. The model parameters are optimized through complementary contrastive loss, thus avoiding the negative impact of sample partitioning errors at the root. Some schemes also combine active learning mechanisms to alleviate the underfitting problem caused by weak supervision signals from pure negative samples.

[0030] This paper presents a sample partitioning and uncertainty modeling method based on evidence-based deep learning. This method transforms the cross-modal similarity of image-text pairs into evidence vectors, and uses the Dirichlet distribution to model the second-order uncertainty of sample matching confidence. It outputs the confidence quality and overall uncertainty of sample matching, and uses this as the basis for separating clean and noisy samples. Some schemes further optimize the sample partitioning logic through energy uncertainty assessment and fuzzy sample thresholding.

[0031] Noise-robust training and auxiliary optimization methods. In addition to the mainstream solutions mentioned above, existing technologies have also derived a variety of auxiliary optimization techniques, including estimating unbiased confidence factors through bidirectional similarity of sample neighborhoods to complete sample weight allocation, classifying fuzzy samples based on inconsistency judgment using dual networks and processing them differently, reassigning semantically similar text labels to noisy images through pseudo-classification tasks, and rematching corresponding content for noisy samples based on optimal transport theory, thereby improving the training effect of the model under noisy data.

[0032] While existing technical solutions have achieved some optimization effects in addressing the noise correspondence problem in image-text matching, they still have many shortcomings and deficiencies in practical applications, as detailed below: Partitioning methods based on probabilistic mixture models suffer from multiple shortcomings in accuracy, efficiency, and robustness. First, to avoid the accumulation of partitioning errors, these methods generally employ dual-network or even multi-network collaborative teaching mechanisms, leading to a significant increase in training time and memory usage, severely limiting scalability in large-scale image and text datasets. Second, they can only achieve hard partitioning of samples, failing to provide quantifiable uncertainty measures for the noise level of samples, resulting in insufficient flexibility and difficulty adapting to complex scenarios involving noise. Third, modeling is based solely on the single-dimensional distribution of sample loss or similarity, neglecting the topological structural association features of samples in the embedding space, leading to a significant bottleneck in partitioning accuracy and a tendency to falsely identify clean samples while missing noisy samples. Fourth, in extreme scenarios with high noise levels (80% and above), these methods are prone to model convergence issues, demonstrating insufficient adaptability to high-noise environments.

[0033] Complementary learning methods suffer from insufficient supervision signals and theoretical adaptability flaws. Firstly, these methods rely solely on negative samples to construct supervision signals, which are weak and prone to underfitting, especially in the early stages of training or when the training data is small, significantly increasing the difficulty of model convergence. Secondly, the theoretical effectiveness of these methods rests on the assumption of a uniform distribution of negative samples. However, noise distribution in real-world internet scenarios is dynamic and non-uniform. Under such complex conditions, the theoretical guarantee of the method fails, leading to a significant drop in practical application performance. Thirdly, these methods can only serve as auxiliary means for model training and cannot independently complete the entire training process of a noise-robust image-text matching model. Furthermore, when combined with sample partitioning results, the negative impact of partitioning errors cannot be completely avoided.

[0034] Existing evidence-based deep learning methods suffer from two main problems: limited evidence dimensions and unresolved inherent limitations. First, current methods rely solely on cross-modal similarity as evidence, neglecting the topological relationships within the intramodal embedding space. This results in limited evidence dimensions, insufficient precision in segmentation, and inadequate reliability in uncertainty measurement, essentially making them no different from probabilistic model-based segmentation methods. Second, they fail to address the inherent problem of evidence underestimation in evidence-based deep learning. The models do not adequately consider high-uncertainty samples, leading to the loss of rare and long-tail semantic features in the training data, thus limiting the model's generalization ability.

[0035] Existing noise utilization and robust training methods suffer from insufficient data value mining and poor scenario adaptability. Firstly, mainstream soft-labeling and soft-distance optimization methods only passively mitigate the adverse effects of noise samples on model training, failing to deeply mine potential semantic associations and rare semantic attributes within the noise samples. This results in semantic information loss in the training data and easily leads to performance fluctuations during model training. Secondly, existing noise sample re-matching methods require manual setting of core parameters such as some transmission quality and the number of pseudo-classification categories, lacking adaptive adjustment capabilities and failing to adapt to dynamically changing noise distributions in real-world scenarios. This results in insufficient theoretical interpretability and scenario generalization ability. Thirdly, most methods focus only on optimizing the point-to-point similarity between images and text, without coordinating optimization of the global topology of the embedding space. This makes it difficult to ensure feature consistency of semantically identical samples in the embedding space, hindering further improvement in image-text matching accuracy. Fourthly, some solutions improve accuracy through multi-network stacking and complex multi-level loss nesting, leading to problems such as high difficulty in hyperparameter tuning, high deployment thresholds, and poor real-time inference, making them unsuitable for lightweight, real-time-critical industrial applications.

[0036] This invention provides a noise-robust multimodal image-text matching method based on multi-evidence fusion, which can be applied to multimodal image-text matching processes in scenarios such as cross-modal retrieval and visual question answering.

[0037] This application addresses the core technical deficiencies of existing technologies in noisy scenarios for multimodal image-text matching tasks, aiming to solve the following technical problems to comprehensively improve the noise robustness, matching accuracy, and engineering adaptability of image-text matching models: 1. This addresses the problem that existing sample segmentation techniques have a single dimension of evidence and cannot provide a quantifiable and reliable measure of second-order uncertainty. It also overcomes the shortcomings of existing schemes, such as the segmentation error accumulating and amplifying during training and the inability of the model to converge in high-noise scenarios.

[0038] 2. To address the problem that existing technologies do not fully mine the potential semantic value of noisy samples, and to overcome the shortcomings of existing rematching strategies, such as reliance on manual setting of core parameters, lack of adaptive capability, inability to adapt to dynamic noise distribution, and easy loss of rare semantic features, thereby improving the effective utilization rate of weakly aligned image and text data.

[0039] 3. To address the inherent problem of evidence underestimation in existing deep learning methods, overcome the shortcomings of unbalanced training weights for high-uncertainty samples and insufficient consistency in bidirectional cross-modal matching of images and text in existing technologies, achieve dynamic balance of training weights for samples with different confidence levels, and enhance the linkage consistency of bidirectional matching.

[0040] 4. Add a global topology optimization loss for the image-text embedding space to address the problem that existing solutions only focus on optimizing the point similarity between images and text, resulting in insufficient feature consistency of samples with the same semantic meaning in the embedding space, which ultimately limits the improvement of matching accuracy.

[0041] 5. To address the issues of complex model architecture and high training and deployment costs in existing technologies, a single network and one forward propagation are used to overcome the shortcomings of existing solutions, such as reliance on multi-network collaborative teaching architecture, high computing power and memory consumption, long training cycles, and difficulty in hyperparameter debugging.

[0042] See Figure 1 This is a flowchart illustrating a noise-robust multimodal image-text matching method based on multi-evidence fusion, provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, the method mainly includes the following steps: Step 1: Obtain multimodal image-text sample pairs, wherein the multimodal image-text sample pairs include images and corresponding text descriptions; Step 2: Input the multimodal image-text sample pairs into the basic matching model to extract the first modal features of the image and the second modal features of the text description; Step 3: Based on the first modal features and the second modal features, extract cross-modal similarity evidence and same-modal topological structure evidence for each multimodal image-text sample; Step 4: Input cross-modal similarity evidence and same-modal topological structure evidence into the evidence fusion module, and fuse them according to the improved Dempster-Shafer combination rule to obtain fused evidence and corresponding uncertainty measure; Step 5: After the warm-up phase of model training is completed, the sample pairs in the training set are divided based on the uncertainty metric to obtain a clean sample subset and a noisy sample subset. Step 6: In the first stage of model training, the parameters of the basic matching model are updated using a clean sample subset, and the global topology of the embedding space of the image and text descriptions is optimized collaboratively using topology preservation loss. Step 7: In the second stage of model training, adaptive re-matching is performed on the subset of noisy samples based on the partially optimal transmission model to mine the potential semantic associations in the subset of noisy samples, and the parameters of the basic matching model are updated using the re-matching results. Step 8: After the model training is completed, obtain the target image-text sample pair, input the target image-text sample pair into the updated base matching model, calculate the cross-modal similarity between the image and the text description in the target image-text sample pair, and output the image-text matching result of the target image-text sample pair based on the cross-modal similarity.

[0043] In practical implementation, this invention addresses the noise-based correspondence problem in multimodal image-text matching tasks by proposing an end-to-end technical solution for noise-robust image-text matching based on multi-evidence fusion. The overall architecture follows a multi-dimensional evidence extraction approach, uncertainty modeling of multi-evidence fusion, robust loss function optimization for clean sample-specific training, and adaptive re-matching of noisy samples. It is compatible with mainstream image-text matching models and addresses the core shortcomings of existing technologies, such as insufficient sample segmentation accuracy, weak noise robustness, inadequate semantic value mining, and high training and deployment costs.

[0044] First, select a specific image-text matching model and image-text data to complete steps 1 and 2. In step 3, extracting two evidence vectors for each multimodal image-text pair based on the image-text features obtained from the image-text matching model requires the following multidimensional evidence extractor.

[0045] Multidimensional Evidence Extractor Design: To address the shortcomings of existing deep learning methods that rely solely on cross-modal similarity as the source of evidence, neglecting the topological structural association features of samples within the modal embedding space, resulting in limited evidence dimensions, incomplete representation of sample matching states, and consequently, insufficient sample segmentation accuracy and poor reliability of uncertainty measurement; and to address the issue that existing probabilistic hybrid models rely solely on sample loss or single-dimensional similarity information for clustering, making them susceptible to training fluctuations and lacking robustness in segmentation results, this invention designs a dual-dimensional multi-source evidence extractor. This extractor simultaneously extracts two independent and effective types of evidence: cross-modal similarity evidence and intramodal topological structural similarity evidence, providing a comprehensive feature basis for judging sample matching states.

[0046] Most techniques only use conventional cross-modal similarity evidence, that is, by using the encoder of the basic image-text matching model to output the embedding vectors of the image and the text, calculate the bidirectional cross-modal matching similarity of all image-text pairs in a single batch, and achieve accurate representation of the point-to-point semantic matching strength of a single sample.

[0047] Specifically, let the training set be... Each sample From the image With text Composition; Cross-modal similarity is calculated using the encoder-similarity paradigm as follows: ; in and These represent the visual and text encoders, respectively. This is the similarity function of the base model (such as cosine similarity or a learnable parameter model). When noisy correspondences exist in the training set, some pairings are incorrect, leading to unreliable supervision signals. Therefore, an uncertainty estimation mechanism needs to be introduced to distinguish between clean and noisy samples.

[0048] To extract multi-level evidence, this invention designs two sources of evidence. In addition to the aforementioned evidence, it adds an embedding structure judgment perspective, namely, intramodal topological structure similarity evidence. Using clean samples identified in the previous training round as structural base points, it measures the neighborhood distribution distance of a single image in the image embedding space and a single text in the text embedding space, respectively, constructing an intramodal neighborhood topological association similarity vector. This vector characterizes the position and structural distribution features of samples in the global embedding space, supplementing the limitations of cross-modal point-to-point similarity perspectives. Specifically, using clean samples identified in the previous training round as structural base points, the structure vector is calculated, i.e.: ; in The similarity calculation function for the basic model. and These are images and text that were identified as clean samples in the previous training round, respectively. Finally, the cosine similarity between the two topological vectors is calculated: ; The last two similarity methods use a consistent nonlinear transformation, outputting synchronous but independent evidence of same-modal topology and cross-modal similarity without interference: ; in This refers to temperature parameters. Within the same batch... In the nth sample, the nth The bidirectional evidence vectors for each sample pair are as follows: .

[0049] In subsequent steps, addressing the limitations of existing technologies in effectively fusing multi-dimensional evidence, providing sufficiently reliable second-order uncertainty measures, and ensuring adequate sample segmentation reliability, this invention introduces subjective logic theory and an improved Dempster-Shafer combination rule to fuse two-dimensional evidence, thereby improving the reliability of subsequent sample segmentation and model training. Simultaneously, addressing the evidence underestimation problem inherent in traditional evidence-based deep learning's Bayesian risk loss, which insufficiently addresses high-uncertainty samples and fails to adapt to the dynamic adjustment requirements of sample weights in noisy scenarios, ultimately leading to overconfidence fitting and insufficient generalization ability, this invention introduces the Fisher Information Matrix (FIM) to weight and optimize traditional evidence loss, designs a composite robust loss function, and incorporates an exchange-weighting mechanism to enhance cross-modal bidirectional matching consistency.

[0050] In step 4, before fusing the two types of evidence, uncertainty needs to be quantified using the standard evidence learning method below, thereby obtaining the trust quality and uncertainty measure, and simultaneously obtaining the specific Dirichlet distribution parameters: 1) Uncertainty quantification based on Dirichlet distribution: like Figure 4 As shown, this invention transforms the evidence vector into an interpretable uncertainty measure and models the second-order uncertainty of the sample's matching confidence. Taking the evidence vector from image to text as an example, the specific implementation is as follows: First, calculate the quality of trust. With overall uncertainty : ; in This represents the j-th dimension of the evidence vector corresponding to the i-th image / text sample in a batch. This is the training batch size.

[0051] Adding 1 to the evidence vector yields the shape parameters of the Dirichlet distribution, thus mapping the evidence to a probability distribution. ; Based on the above parameters, a Dirichlet distribution is constructed to model the sample matching probability: ; corresponding vector It refers to the random variable modeled by the Dirichlet distribution, and each component of it... Let represent the matching probability between the i-th image (text) and the j-th text (image), with expected probability and variance as follows: ; Through the above modeling, the expected confidence level and fluctuation range of sample matching can be output simultaneously, realizing the quantitative characterization of second-order uncertainty and solving the defect that existing hard partitioning methods cannot quantify the degree of sample noise.

[0052] In step 4, combining the trust quality and uncertainty measure yields the subjective opinions corresponding to different pieces of evidence. The fusion of the two types of evidence relies on the following subjective opinion fusion rules: 2) Improved Dempster-Shafer rule for multi-evidence fusion To achieve effective fusion of cross-modal similarity evidence and intramodal topological structure evidence, this invention simplifies the traditional Dempster-Shafer combination rule and adds an adaptive correction mechanism for conflict metrics to avoid semantic conflict distortion in the multi-evidence fusion process. The specific implementation is as follows: Subjective opinions corresponding to cross-modal similarity evidence and subjective opinions corresponding to same-modal topological structure evidence. and The subjective opinion here is A dimensional vector, obtained by combining a trust quality vector and an uncertainty metric, is represented by the fused computation as follows: The combined trust quality and uncertainty are as follows: ; Conflict metric As can be seen from the methods for calculating trust quality and uncertainty, this method can reverse the evidence vector based on the fused subjective opinions, and then calculate the shape parameters of the Dirichlet distribution corresponding to the fused evidence through standard methods, and finally obtain the fused probability distribution function, thereby realizing the complementary fusion of multi-dimensional evidence and comprehensively improving the accuracy of sample segmentation and the reliability of uncertainty measurement.

[0053] During the training process in steps 6 and 7, the loss function required for training includes the following robust loss function based on the Fisher information matrix: 3) Robust loss function design based on Fisher information matrix Specifically, first, the Fisher information matrix of the Dirichlet distribution obtained above is calculated to measure the semantic information abundance of each image and text sample, providing a basis for loss weighting.

[0054] Traditional evidence-based deep learning uses a Bayesian risk loss function designed for training the model as follows: ; in Let be the label vector, representing the prior distribution where the value is 1 only at the i-th position. Let it be the value at its j-th position; As can be seen, the model treats all samples equally. To enhance the model's attention to samples with high uncertainty and to address the evidence underestimation problem in the Bayesian risk loss of traditional evidence deep learning, this study further introduces the Fisher Information Matrix (FIM) to perform weighted optimization of the loss function.

[0055] Specifically, first, the Fisher information matrix of the above distribution is calculated to measure the semantic information abundance of each image-text sample, providing a basis for loss weighting. For a given image-text pair... The neural network outputs an evidence vector. And calculate the Dirichlet distribution parameter vector. For this Dirichlet distribution, the Fisher information matrix (FIM) is calculated as follows: ; in It is a three-gamma function. The sum of the parameters of the Dirichlet distribution described above. Represents a K-dimensional column vector of all 1s. Let represent a K×K matrix of all 1s.

[0056] Assuming soft labels for image-to-text matching If the maximum likelihood estimation follows a Gaussian distribution centered at the Dirichlet mean and with covariance of FIM, it can be proven that the objective of the maximum likelihood estimation can be expressed by a series of scaling and estimation equivalents to minimizing the following loss function: ; The specific implementation and function of each item are as follows: FIM Weighted Mean Square Error Term By weighting the traditional Bayesian risk loss using the Fisher information matrix, the model focuses more on the prediction accuracy of samples with low evidence and high uncertainty, thus fundamentally alleviating the problem of evidence underestimation. The calculation formula is as follows: ; FIM Regularization This is used to constrain the magnitude of the model's evidence output, preventing overconfidence issues caused by excessive evidence in some categories, and improving the model's generalization ability. Its calculation formula is as follows: ; KL divergence regularization term This is used to constrain the predicted distribution to approximate the prior distribution, suppress the accumulation of invalid evidence from negative samples, and further improve the generalization performance of the model. Its calculation formula is as follows: ; Furthermore, this invention introduces an exchange weighting mechanism, which cross-applies the weights of the Fisher information matrices calculated in both the image-to-text and text-to-image directions to the loss function of the other, thereby forcibly strengthening the logical consistency of cross-modal bidirectional matching and reducing the matching accuracy deviation between the two directions.

[0057] In subsequent steps, existing sample partitioning methods cannot achieve accurate hard partitioning based on uncertainty, which easily leads to noisy samples interfering with early model training and causing gradient shifts. Existing methods also lack the ability to collaboratively optimize the embedding space topology and have insufficient feature consistency with semantic samples. This invention achieves more rigorous sample partitioning based on uncertainty measurement of fused evidence output, adapts to the confidence differences of clean samples, and designs a robust training loss system adapted to clean samples. This completely blocks the interference of noisy samples in the early stages of model training and strengthens the basic semantic alignment capability of the model.

[0058] In step 6, after the model completes the preheating training, it enters the early training stage. In order to completely avoid the interference of noisy samples on the gradient direction, this invention only uses clean samples for parameter updates in this stage.

[0059] Therefore, in step 5, for each sample pair, the joint trust quality b output by the evidence deep learning and the overall uncertainty u are used to strictly divide the samples in each mini-batch into clean and noisy samples, thereby obtaining hard labels. Let p be the uncertainty threshold; only samples greater than this threshold are considered strictly clean samples, i.e.: ; in and Let be the values ​​of the k-th dimension of the trust quality vector calculated from the bidirectional evidence vector corresponding to the i-th sample pair. The average value of the uncertainty measure calculated for the bidirectional evidence vector corresponding to the i-th sample pair.

[0060] In this phase, only the clean sample set It participates in subsequent gradient backpropagation. For For clean samples in the dataset, this invention uses asymptotic triplet loss. Specifically, given an anchored clean pair... ,but: ; in For fixed margin parameters, The number of dynamically difficult negative samples is dynamically adjusted with each training round to ensure that the learning of clean samples is gradually strengthened as the training progresses.

[0061] Meanwhile, to ensure the topological similarity of clean image-text pairs in the embedding space, this invention introduces cross-consistency loss and metric consistency loss to collaboratively optimize the global structure of the embedding space and enhance the feature consistency of samples with the same semantic meaning. ; ; in A clean subset of the current mini-batch. It is a fixed margin hyperparameter.

[0062] In summary, the overall loss for each iteration step depends only on clean samples, and the combined loss is as follows: = + + ; in These are the balancing parameters.

[0063] Finally, the overall loss is weighted and combined with the robust evidence loss based on the Fisher information matrix to complete the parameter update of the first stage of the model, thus achieving efficient and robust training of clean samples.

[0064] In step 7, as Figure 3 As shown, existing noise rematching methods require manual setting of core parameters, lack adaptive capabilities, cannot adapt to dynamic noise distributions, and have poor theoretical interpretability. Existing methods can only passively weaken noise interference and cannot fully explore the latent semantics of noise samples, easily leading to the loss of rare semantics. Furthermore, the rematching process easily interferes with the effective features already learned by clean samples. This invention adopts an imbalanced optimal transport model and designs an adaptive rematching strategy for noise samples without manual parameter intervention, fully exploring the latent semantics and rare attributes in noise samples, and further improving the generalization ability of the model.

[0065] Specifically, after completing a set number of rounds of training with clean samples, the noise subset obtained based on uncertainty partitioning is re-matched. The steps are as follows: Cost matrix construction: Assume that a batch of training data has been divided into clean subsets based on uncertainty-based partitioning. With noise subset .

[0066] In a batch of all sample pairs, for clean data It is forcibly defined as 1, the rest Defined as 0. The cost matrix for constructing optimal transmission based on the similarity matrix is: .

[0067] Construction and Solution of the Adaptive Partially Optimal Transmission Problem: This invention constructs a partially optimal transmission problem with regularization constraints. It eliminates the need for manually setting transmission quality thresholds and can adaptively adapt to scenarios with varying noise levels. The problem is defined as follows: ; in For the cost matrix, Denotes the Frobenius inner product of matrices. For entropy regularization, The balancing parameter for the entropy regularization term. The marginal regularization equilibrium parameter, and Let n and m be unit vectors, respectively. and Let be m-dimensional and n-dimensional uniform marginal distribution vectors, respectively, where n and m are both equal to the batch size K, and T is the transfer matrix to be solved. After solving for the transmission plan T using the generalized Sinkhorn algorithm, based on the obtained optimal transmission plan T, the labels i of the noisy samples are normalized to obtain the matching probability vectors of image-to-text and text-to-image pairs for a batch of data pairs. and : ; in To find the value in the i-th row and j-th column of the transmission plan T, and Let represent the vectors in the i-th row and i-th column of the calculated transmission plan T, respectively.

[0068] The rematch loss is defined as the symmetric KL divergence. The model is optimized by comparing the difference between the rematch probability obtained through the transmission plan T and the original matching probability, where the bidirectional original matching probability... and This can be obtained from the expectations of the Dirichlet distributions corresponding to the two-way evidence vectors: ; The rematch loss is weighted and combined with the robust evidence loss based on the Fisher information matrix to complete the parameter update of the second stage of the model, so as to realize the semantic reuse of noisy samples and the correction of matching relationships.

[0069] This embodiment presents a noise-robust multimodal image-text matching method based on multi-evidence fusion. By designing a dual-dimensional multi-source evidence extractor, it simultaneously extracts cross-modal similarity evidence and intramodal topological structure evidence. An improved Dempster-Shafer combination rule is then used to fuse these two types of evidence, compensating for the incomplete information of single-dimensional evidence. This achieves accurate segmentation between clean and noisy samples, while outputting quantifiable and traceable sample matching uncertainty, effectively overcoming the convergence problem of existing technologies in high-noise environments. Furthermore, by introducing cross-consistency loss and metric consistency loss, the global topological structure of the image and text embedding spaces is collaboratively optimized, strengthening the feature consistency of semantically similar samples in different modal embedding spaces. This significantly improves the matching performance under full-gradient noise conditions. The comprehensive index of image-text matching effectively suppresses the performance degradation in high-noise scenarios, thereby significantly enhancing the model's noise robustness. By adopting a single-network end-to-end architecture, training can be completed with only one forward propagation and one backward propagation, eliminating the need for multi-network collaborative mechanisms. This greatly simplifies the model architecture and training process, significantly reducing training memory usage and deployment costs, making this solution more suitable for industrial deployment in large-scale image-text data scenarios. By employing an imbalanced optimal transmission model for adaptive semantic re-matching of noisy samples, there is no need to manually set core parameters such as transmission quality. This allows for adaptive adaptation to different noise distribution scenarios, fully mining the latent semantics and rare attributes in noisy samples, avoiding the loss of rare semantic features, and thus significantly improving the model's data utilization and generalization ability.

[0070] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof.

[0071] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A noise-robust multimodal image-text matching method based on multi-evidence fusion, characterized in that, include: Step 1: Obtain multimodal image-text sample pairs, wherein the multimodal image-text sample pairs include images and corresponding text descriptions; Step 2: Input the multimodal image-text sample pairs into the basic matching model to extract the first modal features of the image and the second modal features of the text description; Step 3: Based on the first modal features and the second modal features, extract cross-modal similarity evidence and same-modal topological structure evidence for each multimodal image-text sample; Step 4: Input cross-modal similarity evidence and same-modal topological structure evidence into the evidence fusion module, and fuse them according to the improved Dempster-Shafer combination rule to obtain fused evidence and corresponding uncertainty measure; Step 4 specifically includes: First subjective opinion corresponding to cross-modal similarity evidence Second subjective opinion corresponding to the same modal topological structure evidence An improved Dempster-Shafer combination rule with an adaptive correction mechanism incorporating conflict metrics is used for fusion, and the trust quality of the fused result is... and uncertainty measurement for: in, , All for A dimensional vector of subjective opinions. As a conflict metric, it is calculated using the conflict trust quality from two subjective opinion vectors. This represents the k-th dimension of trust quality in the first subjective opinion vector. This represents the k-th dimension of trust quality in the second subjective opinion vector. This represents the uncertainty in the first subjective opinion vector. This represents the uncertainty in the second subjective opinion vector; Based on the quality of trust after integration and the uncertainty after fusion Restore the fused evidence vector; The fused evidence vector is incremented by 1 to obtain the shape parameter of the Dirichlet distribution. The Dirichlet distribution is then constructed based on the shape parameter to model the sample matching probability. Calculate the trust quality based on the recovered and fused evidence vector. and overall uncertainty : in, This is the evidence vector obtained after fusion and reconstruction. For the total amount of evidence, Let be the dimension of the evidence vector; Step 5: After the warm-up phase of model training is completed, the sample pairs in the training set are divided based on the uncertainty metric to obtain a clean sample subset and a noisy sample subset. Step 6: In the first stage of model training, the parameters of the basic matching model are updated using a clean sample subset, and the global topology of the embedding space of the image and text descriptions is optimized collaboratively using topology preservation loss. Step 7: In the second stage of model training, adaptive re-matching is performed on the subset of noisy samples based on the partially optimal transmission model to mine the potential semantic associations in the subset of noisy samples, and the parameters of the basic matching model are updated using the re-matching results. Step 8: After the model training is completed, obtain the target image-text sample pair, input the target image-text sample pair into the updated base matching model, calculate the cross-modal similarity between the image and the text description in the target image-text sample pair, and output the image-text matching result of the target image-text sample pair based on the cross-modal similarity.

2. The method according to claim 1, characterized in that, Step 3 specifically includes: Calculate the bidirectional cross-modal matching similarity between the image and the text description in a multimodal image-text sample pair to obtain the cross-modal similarity matrix; Using the identified clean samples as structural base points, we calculate the neighborhood distribution distance of a single image in the image embedding space and the neighborhood distribution distance of a single text description in the text embedding space, and construct the image intramodal similarity matrix and the text intramodal similarity matrix. Calculate topological similarity based on the intra-modal similarity matrix of images and the intra-modal similarity matrix of text; The cross-modal similarity and topological similarity in the cross-modal similarity matrix are transformed independently into cross-modal similarity evidence and same-modal topological evidence through nonlinear transformation.

3. The method according to claim 2, characterized in that, The step of calculating topological similarity based on the image intra-modal similarity matrix and the text intra-modal similarity matrix includes: Construct an image topology vector based on the image intra-modal similarity matrix and the text intra-modal similarity matrix. and text topology vector ; Calculate the cosine similarity between the image topological structure vector and the text topological structure vector, and use this as the topological similarity. .

4. The method according to claim 1, characterized in that, The first stage of training the basic matching model employs a robust loss function based on the Fisher information matrix, which includes: Fisher information matrix weighted mean square error term This is used to weight the traditional Bayesian risk loss, enhancing the model's attention to low-evidence sample pairs; Fisher Information Matrix Regularization Term , used to constrain the magnitude of the model's evidence output; KL divergence regularization term This is used to constrain the predicted distribution to approximate the prior distribution; The expression for the robust loss function is: in, , These are the weighting coefficients.

5. The method according to claim 4, characterized in that, The weighted mean square error term of the Fisher information matrix The expression is: in, It is a three-gamma function. The first of the Dirichlet distributions The parameters are calculated from the evidence vector. The sum of the parameters of the Dirichlet distribution; The Fisher information matrix regularization term The expression is: 。 6. The method according to claim 1, characterized in that, The topology preservation loss includes cross-consistency loss and metric consistency loss: The cross-consistency loss The expression for constraining the bidirectional cross-modal similarity between image-to-text and text-to-image is: The measurement of consistency loss The expression used to constrain the consistency of intramodal similarity and cross-modal similarity is: in, For the clean sample subset in the current batch, They are the same margin hyperparameter.

7. The method according to claim 1, characterized in that, The step of adaptively rematching a subset of noise samples based on a partially optimal transmission model includes: Based on clean and noisy sample subsets, a cross-modal similarity matrix is ​​constructed, and a cost matrix is ​​constructed based on the cross-modal similarity matrix: in Cross-modal similarity between sample pairs; Constructing a partially optimal transport problem with regularization constraints: in, For the transmission cost matrix, Denotes the Frobenius inner product of matrices. For entropy regularization, The balancing parameter for the entropy regularization term. The marginal regularization equilibrium parameter, and Let n and m be unit vectors, respectively. and Let be m-dimensional and n-dimensional uniform marginal distribution vectors, respectively, and T be the transfer matrix to be solved. The generalized Sinkhorn algorithm is used to solve a partial optimal transmission problem, resulting in a transmission plan T. The rematch probability is calculated based on the transmission plan T, and the parameters of the basic matching model are updated using the symmetric KL divergence as the rematch loss.

8. The method according to claim 1, characterized in that, In the first stage of training the basic matching model, an incremental triplet loss is used to optimize the clean sample subset. The number of extremely difficult negative samples w in the incremental triplet loss decreases adaptively with each training round. in, For batch size, The attenuation rate, This represents the minimum number of difficult cases.

9. The method according to claim 1, characterized in that, During the warm-up phase, the basic matching model is trained using all training samples without dividing or removing noisy sample pairs. The number of training cycles during the warm-up phase is adaptively determined based on the preset noise ratio.

Images